Map construction method and device, equipment and storage medium

Abstract

This invention discloses a map construction method, apparatus, device, and storage medium. The method includes: acquiring sensor data collected by a preset sensor; establishing pose constraints for a movable object and semantic constraints for a semantic object based on the sensor data; performing joint optimization based on the pose and semantic constraints to determine the semantic result of the semantic object and the pose result of the movable object; and constructing a semantic map and a point cloud map based on the semantic and pose results. By adopting the above technical solution, the construction process of the semantic map and the point cloud map are integrated, and the pose and semantic constraints are used for joint optimization, enabling the semantic and pose results to form a mutual constraint relationship. This allows for the rapid and accurate acquisition of the semantic map and point cloud map, resulting in a more precise high-precision map.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of high-precision map technology, and in particular to map construction methods, apparatus, devices and storage media. Background Technology

[0002] With the development of autonomous driving and intelligent robot technologies, ensuring the precise movement of autonomous vehicles and intelligent robots has become a hot topic. High-definition maps are commonly used in autonomous driving technology, primarily including point cloud maps and semantic maps. Point cloud maps can be three-dimensional (3D) point clouds containing spatial location information, mainly used for registration-based online positioning. Semantic maps can be maps including road information such as lane lines, poles, or traffic lights, and can be used for online positioning and path planning. Traffic elements such as lane lines, poles, or traffic lights can be referred to as semantic objects.

[0003] Currently, there are two main methods for constructing high-precision point cloud maps and lane line maps: The first method utilizes high-precision integrated navigation equipment to calculate real-time pose, then converts the point cloud obtained from LiDAR scanning to the world coordinate system based on the pose. Accumulating data over a period of time or distance yields a dense point cloud map. Subsequently, based on the different reflectivities of different objects in the point cloud map, a semantic map is manually drawn. The second method uses traditional Simultaneous Localization and Mapping (SLAM) to achieve pose estimation and sparse point cloud map construction. Based on this, semantic segmentation is performed on the two-dimensional (2D) image to obtain points related to road information, thereby constructing a semantic map.

[0004] However, existing methods for manually drawing lane lines are costly and inefficient. Furthermore, the reflectivity of point clouds is easily affected by external factors such as road surface material, humidity, and the installation height of LiDAR sensors. In real-world road scenarios, it is difficult to accurately identify all road information based solely on point cloud reflectivity. Visual methods, on the other hand, directly obtain 2D information, resulting in lower accuracy and larger errors in the constructed semantic maps. Therefore, existing high-precision map construction schemes are still imperfect and require improvement. Summary of the Invention

[0005] The embodiments of the present invention provide a map building method, apparatus, device and storage medium, which can optimize existing high-precision map building schemes.

[0006] In a first aspect, embodiments of the present invention provide a map construction method, including:

[0007] Acquire sensor data collected by preset sensors;

[0008] Based on the sensor data, establish the pose constraint relationship of the movable object and the semantic constraint relationship of the semantic object;

[0009] A joint optimization solution is performed based on the pose constraints and the semantic constraints to determine the semantic result of the semantic object and the pose result of the movable object; and

[0010] A semantic map and a point cloud map are constructed based on the semantic results and the pose results.

[0011] In a second aspect, embodiments of the present invention provide a map building apparatus, comprising:

[0012] The data acquisition module is used to acquire sensor data collected by preset sensors;

[0013] The constraint relationship establishment module is used to establish the pose constraint relationship of the movable object and the semantic constraint relationship of the semantic object based on the sensor data.

[0014] The joint solution module is used to perform joint optimization solution based on the pose constraint relationship and the semantic constraint relationship to determine the semantic result of the semantic object and the pose result of the movable object;

[0015] The map building module is used to build a semantic map and a point cloud map based on the semantic results and the pose results.

[0016] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the map building method provided in embodiments of the present invention.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the map construction method provided in embodiments of the present invention.

[0018] The map construction scheme provided in this embodiment of the invention establishes pose constraints for movable objects and semantic constraints for semantic objects based on sensor data collected by preset sensors. It then performs joint optimization based on these pose and semantic constraints to determine the semantic results and the pose results of the movable objects. Finally, it constructs a semantic map and a point cloud map based on the semantic and pose results. By adopting the above technical solution, the construction processes of the semantic map and the point cloud map are integrated. The joint solution using pose and semantic constraints allows the semantic and pose results to form mutual constraints, enabling the rapid and accurate acquisition of semantic and point cloud maps, resulting in a more precise high-precision map and improving the autonomous movement performance of movable objects. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a map construction method provided in an embodiment of the present invention;

[0020] Figure 2 A flowchart illustrating another map construction method provided in an embodiment of the present invention;

[0021] Figure 3 A schematic diagram illustrating the principle of a map construction process provided in an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the map construction result of the bridge scene provided in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the map construction result of a tunnel scene provided in an embodiment of the present invention;

[0024] Figure 6 A structural block diagram of a map building device provided in an embodiment of the present invention;

[0025] Figure 7 This is a structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] To enable those skilled in the art to better understand this application, some technical terms appearing in the embodiments of this application are explained below:

[0029] Mobile objects refer to objects such as vehicles, mobile robots, and aircraft that can be used for map collection. Various types of sensors, such as lidar and cameras, can be mounted on mobile objects.

[0030] Simultaneous localization and mapping: In an unknown environment, a movable object can locate itself based on position estimation and a map, and simultaneously build a map incrementally based on its own location.

[0031] Mapping: Based on the estimated real-time pose of vehicles or mobile robots and the data collected by visual sensors such as LiDAR, a high-precision map describing the current scene is constructed.

[0032] Pose: A general term encompassing position and orientation, comprising six degrees of freedom, including three positional degrees of freedom and three orientation degrees of freedom. The three orientation degrees of freedom are typically represented by pitch, roll, and yaw.

[0033] Frame: The measurement data received by a sensor after completing one observation. For example, a frame of data from a camera is an image, and a frame of data from a lidar is a set of laser point clouds.

[0034] Registration: Matching observations of the same region at different times and locations to obtain the relative pose relationship between the two observation times.

[0035] Nonlinear optimization: an optimization problem that solves a set of equations and inequalities consisting of a series of unknown real functions.

[0036] Keyframe: When using nonlinear optimization methods or other methods for multi-sensor fusion, in order to reduce the computational load of the system, a keyframe is usually created when the change in distance or attitude exceeds a certain threshold, and the keyframe is generally optimized during optimization.

[0037] GNSS: Global Navigation Satellite System, which may include GPS or BeiDou positioning system, etc.

[0038] GPS: Global Positioning System.

[0039] IMU: Inertial Measurement Unit, is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object.

[0040] Combined navigation equipment: such as combined navigation equipment that includes GNSS and IMU.

[0041] In some embodiments of this application, the term "vehicle" is broadly interpreted to include any moving object, including, for example, aircraft, ships, spacecraft, automobiles, trucks, vans, semi-trailers, motorcycles, golf carts, off-road vehicles, warehouse transport vehicles, or agricultural vehicles, as well as rail transport vehicles, such as trams or trains, and other rail vehicles. "Vehicle" as used in this application typically includes: a power system, a sensor system, a control system, peripheral equipment, and a computer system. In some embodiments, a vehicle may include more, fewer, or different systems.

[0042] Figure 1 This is a flowchart illustrating a map building method provided by an embodiment of the present invention. This method is applicable to high-precision map building applications and can be executed by a map building device, which can be implemented by software and / or hardware, and is generally integrated into a computer device. This computer device can be integrated into a movable object, or it can be externally connected to or communicate with the movable object. Figure 1 As shown, the method includes:

[0043] Step 101: Obtain sensor data collected by the preset sensor.

[0044] For example, the preset sensor can be mounted on the movable object, and the specific mounting method is not limited. It can be integrated into the movable object or externally attached to the movable object. The type of preset sensor is not limited. It can be any sensor used to establish pose constraint relationships. For example, it can include one or more of the following: INS, IMU, wheel speedometer, image acquisition device, point cloud acquisition device, accelerometer, and gyroscope. Some sensors can also be integrated together. For example, the IMU can include accelerometer and gyroscope. GNSS and IMU can be used as a combined navigation device. In some embodiments, the image acquisition device is, for example, a camera, and the point cloud acquisition device is, for example, lidar, millimeter-wave radar, ultrasonic radar, etc., but it is not limited to these.

[0045] Step 102: Establish the pose constraint relationship of the movable object and the semantic constraint relationship of the semantic object based on the sensor data.

[0046] In some embodiments, this disclosure establishes pose constraint relationships for a movable object based on sensor data collected by a first set of sensors, and establishes semantic constraint relationships for a semantic object based on sensor data collected by a second set of sensors. The semantic object is a semantic object within the environment in which the movable object is located. The pose constraint relationship includes pose-related state variables, and the semantic constraint relationship includes semantically related state variables. The first set of sensors may include, for example, a combined navigation device, a wheel speedometer, a laser acquisition device, etc., and may also include an image acquisition device. The second set of sensors includes an image acquisition device and a point cloud acquisition device. This disclosure generates a three-dimensional spatial representation of the semantic object based on the image acquired by the image acquisition device and the point cloud acquired by the laser acquisition device, and obtains the state variables of the semantic object through joint optimization.

[0047] In this embodiment, the specific type of pose constraint relationship is not limited. For example, it can be a global pose constraint relationship or a local consistency constraint relationship. Local consistency constraints may include, for example, IMU pre-integration constraints, velocity constraints, and point cloud registration constraints. Different pose constraint relationships can be established based on the different selections of preset sensors in the movable object, or corresponding preset sensors can be configured for the movable object according to the required pose constraint relationship. When establishing pose constraint relationships, a corresponding error function can be established based on sensor data as the pose constraint relationship. The state variables included in the error function and the specific expression form of the error function are not limited and can be set according to the actual situation.

[0048] A semantic map may contain many types of semantic objects, such as lane lines, poles, signs, curbs, and traffic lights. In this embodiment of the invention, the semantic objects involved in establishing semantic constraint relationships can be one or more, without any specific limitation. It should be noted that when there are multiple semantic objects, they may also include semantic objects of the same type. For example, if the semantic object type is lane lines, multiple lane lines may be involved in establishing semantic constraint relationships.

[0049] When establishing semantic constraint relationships, 2D information in images and 3D information in point clouds can be effectively combined to improve the calculation accuracy of 3D information of semantic objects, which is conducive to building high-precision semantic maps. The specific combination method can be set according to the actual situation. Corresponding error functions can be established based on 2D and 3D information as semantic constraint relationships. The state variables contained in the error function and the specific expression form of the error function are not limited.

[0050] Step 103: Perform joint optimization based on the pose constraint relationship and the semantic constraint relationship to determine the semantic result of the semantic object and the pose result of the movable object.

[0051] For example, a joint error function can be established based on the error function corresponding to the pose constraint relationship and the error function corresponding to the semantic constraint relationship. The joint error function can be solved using a certain solution method to obtain the estimated values ​​of each state variable. The estimated values ​​of the state variables related to semantics are determined as the semantic result, and the estimated values ​​of the state variables related to pose are determined as the pose result of the movable object.

[0052] Step 104: Construct a semantic map and a point cloud map based on the semantic results and the pose results.

[0053] For example, the semantic results can be processed to obtain the 3D contour of the semantic object, and then a semantic map can be constructed based on the 3D contour of the semantic object. A point cloud map can be obtained by overlaying the point cloud based on the pose results.

[0054] The map construction method provided in this embodiment of the invention establishes pose constraints for movable objects and semantic constraints for semantic objects based on sensor data collected by preset sensors. It then performs joint optimization based on these pose and semantic constraints to determine the semantic results and the pose results of the movable objects. Finally, it constructs a semantic map and a point cloud map based on the semantic and pose results. By adopting the above technical solution, the construction process of the semantic map and the point cloud map are integrated. The joint solution using pose and semantic constraints allows the semantic and pose results to form mutual constraints, enabling the rapid and accurate acquisition of semantic maps and point cloud maps, resulting in more precise high-precision maps and improving the autonomous movement performance of movable objects.

[0055] In some embodiments, the sensor data acquisition includes images acquired by an image acquisition device and point clouds acquired by a point cloud acquisition device. Establishing semantic constraint relationships for semantic objects based on the sensor data includes: determining the observed values ​​of the three-dimensional contours corresponding to each semantic object based on the images and the point clouds; and establishing semantic constraint relationships based on the observed values ​​of the three-dimensional contours. The advantage of this approach is that establishing semantic constraint relationships based on the observed values ​​of the 3D contours of semantic objects allows 3D contours to better represent the spatial positional relationships between semantic objects and between semantic objects and movable objects compared to 2D contours. Furthermore, combining images and point clouds can improve the computational accuracy of the 3D information of semantic objects.

[0056] In some embodiments, determining the observed value of the 3D contour corresponding to the semantic object based on the image and the point cloud includes: performing semantic segmentation on the image to obtain a 2D contour of the region where the semantic object is located; projecting the point cloud onto the image and determining the target 3D point corresponding to the semantic object based on the 2D contour; and fitting the target 3D point to obtain the observed value of the 3D contour corresponding to the semantic object. The advantage of this approach is that it yields more accurate 3D contour observed values.

[0057] Semantic segmentation can be achieved using deep learning. Specifically, relevant models can be trained specifically according to the type of semantic object to accurately segment the region where the semantic object is located in the image. Subsequently, 3D points in the point cloud can be projected onto the semantically segmented image using the intrinsic parameters of the image acquisition device and the extrinsic parameters between the image acquisition device and the point cloud acquisition device. Points falling within the 2D contour of the semantic object's region can be identified as target 3D points. After fitting the target 3D points according to the characteristics of the semantic object's 3D contour, the observed value of the 3D contour corresponding to the semantic object is obtained.

[0058] In some embodiments, fitting the target 3D point to obtain the observed value of the 3D contour corresponding to the semantic object includes: performing planar fitting or linear fitting on the target 3D point to obtain the corresponding spatial equation; determining the 3D coordinates of the target 3D point based on the pixel coordinates of the 2D contour, the intrinsic parameters of the image acquisition device, and the extrinsic parameters between the image acquisition device and the point cloud acquisition device; and fitting the target 3D point based on the spatial equation and the 3D coordinates of the target 3D point to obtain the observed value of the 3D contour corresponding to the semantic object. The advantage of this configuration is that it allows for more accurate acquisition of the observed value of the 3D contour corresponding to the semantic object.

[0059] For example, semantic objects can be abstracted into corresponding 3D shapes based on the characteristics of their 3D contours. Different representational quantities can be set for different semantic objects to represent their corresponding 3D contours. For instance, a semantic object with a large surface area and small thickness can be abstracted into a planar figure, and its relative position or angle in space can be represented by a normal vector, thus obtaining a spatial figure. During fitting, planar fitting can be performed. If the thickness is large, it can be abstracted into a cube or similar shape, and planar fitting can be performed on each face separately. Similarly, a cylindrical semantic object can be abstracted into a cylinder; if the radius is small, linear fitting can be performed. Furthermore, it should be noted that the same semantic object may have a complex shape and can be decomposed, for example, into a cylinder and a planar image. In this case, fitting can be performed separately for each component of the semantic object to obtain the corresponding spatial equation. The 3D coordinates of the target 3D point can be determined based on the pixel coordinates of the 2D contour, the intrinsic parameters of the image acquisition device, and the extrinsic parameters between the image acquisition device and the point cloud acquisition device. The observation values ​​of the 3D contour corresponding to the semantic object are obtained by fitting the spatial equation and the 3D coordinates of the target 3D point using a preset analysis method. The preset analysis method can be, for example, Principal Component Analysis (PCA).

[0060] Lane lines are one of the most common semantic objects in semantic maps. In some embodiments, the semantic object includes lane lines, and the three-dimensional contour corresponding to the lane lines includes a spatial rectangle (also called a 3D rectangle). The representation of the spatial rectangle may include at least one of the following: center point, normal vector, long side vector, length, and width. The fitting method for the target three-dimensional point corresponding to the lane line is planar fitting, thereby obtaining the corresponding spatial plane equation. Each dashed lane line corresponds to one spatial rectangle, and each solid lane line corresponds to multiple spatial rectangles based on its length and curvature. The advantage of this setup is that it allows for accurate representation of the 3D contour of the lane lines. Dashed lane lines are generally composed of multiple segments, and the geometric structure of each dashed lane line segment can be approximated as a rectangle. Each dashed line segment can be represented using a spatial rectangle. Solid lane lines are usually longer, and roads often have a certain curvature. To ensure mapping accuracy, long solid lines can be divided into multiple shorter segments based on their length and curvature, that is, multiple consecutive spatial rectangles can be used to represent the solid line. Each segment is fitted separately to obtain the geometric information of multiple consecutive spatial rectangles.

[0061] Road poles (which may include light poles or other pole-shaped road signs) are among the most common road signs in semantic maps of highway scenes. Matching observed road poles with prior maps can effectively improve positioning accuracy. In some embodiments, the semantic object includes a road pole, and the corresponding 3D contour of the road pole includes a cylinder. The representative quantities of the cylinder include at least one of the top center point, the bottom center point, and the radius. The fitting method for the target 3D point corresponding to the road pole is linear fitting, thereby obtaining the corresponding spatial linear equation. The advantage of this setup is that the 3D contour of the road pole can be accurately represented.

[0062] In some embodiments, establishing semantic constraint relationships based on the observations of the 3D contours includes: for each keyframe in a preset sliding window, establishing a correlation between the observations of the 3D contours in the current keyframe and a set of semantic object state variables, so as to update the semantic object state variables in the set of semantic object state variables, and establishing an error function corresponding to each 3D contour in the current keyframe; and establishing semantic constraint relationships according to the error functions corresponding to each keyframe in the preset sliding window. The advantage of this setup is that by fusing the keyframes in the preset sliding window, the mapping accuracy can be effectively improved.

[0063] For example, the size of the preset sliding window can be preset according to the actual situation or dynamically set according to the preset rules. The preset sliding window usually contains multiple keyframes and can maintain a dynamically updated set of semantic object state variables. The three-dimensional contour obtained by fitting each keyframe in the preset sliding window can be associated with the set of semantic object state variables, thereby establishing the error function corresponding to each three-dimensional contour. By integrating these error functions, semantic constraint relationships can be obtained.

[0064] In some embodiments, for lane lines, establishing the association between the observed values ​​of the 3D contour in the current keyframe and the semantic object state variable set to update the semantic object state variables in the semantic object state variable set includes: for each spatial rectangle in the current keyframe, calculating a first distance between the observed value of the center point of the current spatial rectangle and the center point of each rectangle state variable in the lane line state variable set in the same coordinate system; if there is a first distance less than a first preset distance threshold, then the current spatial rectangle is associated with the corresponding rectangle state variable; otherwise, a rectangle state variable corresponding to the current spatial rectangle is created in the lane line state variable set. The advantage of this setting is that it accurately associates the data of the same lane line rectangle in different keyframes. When a new lane line rectangle appears as the movable object moves, a corresponding rectangle state variable is created in the lane line state variable set in a timely manner, ensuring accurate dynamic updates of the lane line state variable set. The first preset distance threshold can be set according to actual needs.

[0065] In some embodiments, for a road pole, establishing the association between the observed values ​​of the 3D contour in the current keyframe and the semantic object state variable set to update the semantic object state variables in the semantic object state variable set includes: for each pole in the current keyframe, calculating the third distance between the observed value of the top center point of the current pole and the top center of each rectangular state variable in the road pole state variable set, and calculating the fourth distance between the observed value of the bottom center point of the current pole and the bottom center of each rectangular state variable in the lane line state variable set. If both the third distance and the fourth distance are less than a second preset distance threshold and a third preset distance threshold are satisfied, then the current pole is associated with the corresponding pole state variable; otherwise, a pole state variable corresponding to the current pole is created in the road pole state variable set. The advantage of this setting is that it accurately associates the data of the same road pole in different keyframes. When a new road pole appears as the movable object moves, the corresponding pole state variable is created in the road pole state variable set in a timely manner, ensuring accurate dynamic updating of the road pole state variable set. The second and third preset distance thresholds can be set according to actual needs.

[0066] In some embodiments, the error function in the current keyframe includes the lane line error function corresponding to the spatial rectangle and / or the road pole error function corresponding to the pillar. The lane line error function includes the error functions of each dashed lane line and each solid lane line. This configuration enriches the types of semantic objects in the semantic map, thereby further improving mapping accuracy.

[0067] In some embodiments, the error function of the dashed lane line includes: error functions relating the state variables of the center point, normal vector, long side vector, length, and width to the observed values, and error functions relating the state variables of the normal vector and long side vector to the unit vector, respectively. The advantage of this configuration is that it allows for a more comprehensive construction of the error function of the dashed lane line from multiple dimensions, which is beneficial for improving mapping accuracy.

[0068] In some embodiments, the error function for solid lane lines includes: error functions relating to the state variables of the normal vector, the long side vector, and the width with respect to the observed values; a second distance from the state variable of the center point to the observed center line; and error functions relating the state variables of the normal vector and the long side vector to the unit vector, respectively. The advantage of this configuration is that, since solid lane lines are continuous, the error function relating to length can be omitted, balancing the comprehensiveness of the error function for solid lane lines with the construction speed, which is beneficial for improving mapping accuracy and controlling the computational load of joint solutions.

[0069] In some embodiments, the pole error function includes: an error function relating the state variables of the top center point, the bottom center point, and the radius to the observed values.

[0070] It should be noted that when constructing the error functions corresponding to the above semantic objects, if the state variables and the observed values ​​are not in the same coordinate system, they can be transformed to the same coordinate system first. The specific coordinate system to which they are transformed can be set according to the actual situation. Generally, it can be consistent with the state variables, such as the world coordinate system.

[0071] In some embodiments, when the pose constraint relationship includes a global pose constraint relationship, establishing the global pose constraint relationship of the movable object based on the collected sensor data includes: estimating the current pose at the current moment based on the pose at the previous moment and the pose change from the previous moment to the current moment, wherein the pose change is obtained through IMU integration; if the difference between the estimated current pose and the pose observation value is less than a preset difference threshold, then establishing the global pose constraint relationship corresponding to the current moment based on the pose state variable and the pose observation value; otherwise, discarding the global pose constraint relationship corresponding to the current moment. The advantage of this setting is that it improves the accuracy of the global pose. Here, the collected sensor data may include sensor data collected by the integrated navigation device, and the pose observation value may be the pose observation value of the integrated navigation device, which may include sensor data collected by GNSS or sensor data collected by IMU. Of course, the collected sensor data may also be data collected by a single sensor, and the pose observation value may also include the pose observation value of a single sensor. Integrated navigation devices are commonly used sensors for mapping movable objects, providing high-precision real-time global pose. However, factors such as occlusion or multipath effects can affect positioning accuracy, and may even lead to positioning failure. Incorrect global positions can severely reduce accuracy. In this embodiment of the invention, the pose at the current moment is calculated based on the pose at the previous moment and the pose change obtained by IMU integration. If the difference between the calculated pose and the pose observed by the integrated navigation device exceeds a set threshold, the observation by the integrated navigation device at that moment is considered inaccurate and is discarded to avoid affecting the accuracy of the global pose.

[0072] In some embodiments, the step of jointly optimizing the solution based on the pose constraint relationship and the semantic constraint relationship to determine the semantic result of the semantic object and the pose result of the movable object includes: constructing a joint optimization objective function corresponding to each keyframe within a preset sliding window based on the pose constraint relationship and the semantic constraint relationship; solving the joint optimization objective function to obtain the estimated values ​​of each state variable; wherein, the estimated values ​​of the semantic object state variables are recorded as the semantic result, and the estimated values ​​of the position state variables and the pose state variables are recorded as the pose result. The advantage of this setting is that the preset sliding window can be used to reasonably determine the data range for joint optimization, balancing mapping accuracy and mapping efficiency. The set of state variables used for joint optimization in this disclosure includes both state variables representing pose and state variables representing semantic objects, thereby simultaneously achieving joint optimization of pose state variables and semantic object state variables during the joint optimization process. The specific solution method is not limited and can be a nonlinear optimization solution method, such as Ceres-Solver.

[0073] In some embodiments, constructing a semantic map and a point cloud map based on the semantic results and the pose results includes: calculating the corner points of a semantic object based on the semantic results and geometric relationships; connecting the corner points to obtain a 3D contour of the semantic object; combining the 3D contours of the semantic objects to obtain a semantic map of the semantic object; and overlaying the point cloud based on the pose results to obtain a point cloud map. Specifically, for dashed lane lines, the 3D contour corresponds to the four corner points of a spatial rectangle connected sequentially; for solid lane lines, the 3D contour corresponds to the two side lines obtained by connecting the corner points in the length direction in spatial order. The advantage of this configuration is that it effectively utilizes the semantic results and pose results to quickly and accurately construct semantic maps and point cloud maps, which are then output to downstream modules.

[0074] In some embodiments, semantic maps and point cloud maps can be integrated into a single map, such as overlaying a semantic map onto a point cloud map to present the aforementioned semantic results, thereby improving the visualization and convenience of map presentation and reducing the amount of data.

[0075] In some embodiments, the point cloud registration constraints are obtained as follows: For the point clouds of each keyframe within a preset sliding window, each point is divided into line feature points and planar feature points based on the curvature information of the points in the point cloud; for each keyframe within the preset sliding window, the pose of the previous keyframe is superimposed with the inter-frame pose change obtained by IMU integration to obtain the initial registration pose corresponding to the current keyframe; for line feature points and planar feature points, nearest neighbor search is performed in the local map based on the initial registration pose to obtain the corresponding nearest neighbor feature point set, wherein the local map includes the feature points of each keyframe in the preset sliding window; line error function and planar error function are established based on the corresponding nearest neighbor feature point set; and point cloud registration constraints are determined based on the line error function and planar error function within the preset sliding window. The advantage of this setting is that the frame-to-local map registration method can improve the accuracy and efficiency of point cloud registration, and since the registration is performed on keyframes within the preset sliding window, it can be associated with the establishment of semantic constraint relationships and cooperate with subsequent joint optimization solutions.

[0076] Optionally, before establishing point cloud registration constraints, the point cloud data can be preprocessed according to the characteristics of the point cloud acquisition device used. For example, the observation data of a mechanical lidar may be subject to motion distortion due to the movement of movable objects. For each frame of lidar data, motion compensation can be performed on the point cloud first.

[0077] In some embodiments, a line error function is established based on the nearest neighbor feature point set corresponding to a line feature point, including: performing line fitting based on the nearest neighbor feature point set corresponding to the line feature point, and determining the line error function based on the distance from the line feature point to the corresponding fitted line. A surface error function is established based on the nearest neighbor feature point set corresponding to a planar feature point, including: performing planar fitting based on the nearest neighbor feature point set corresponding to the planar feature point, and determining the surface error function based on the distance from the planar feature point to the corresponding fitted plane. The advantage of this approach is that it accurately constructs corresponding error functions based on the characteristics of both line and planar feature points, further improving mapping accuracy.

[0078] Figure 2 This is a flowchart illustrating another map construction method provided in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the principle of a map construction process provided in an embodiment of the present invention, which can be combined with... Figure 2 and Figure 3 To facilitate understanding of the embodiments of the present invention, a vehicle is used as an example of a movable object.

[0079] like Figure 2 As shown, the method may include:

[0080] Step 201: Establish the vehicle's pose constraint relationship based on the sensor data collected by the preset sensors.

[0081] In this embodiment of the invention, the preset sensors may include a combined navigation device (such as GNSS / IMU), an inertial measurement unit (IMU), a wheel speedometer, and a lidar. The IMU may include an accelerometer and a gyroscope, and the IMU may be integrated into the combined navigation device. Since the observations from multiple preset sensors are fused, it may involve an IMU coordinate system (denoted as I), a vehicle coordinate system (denoted as B), a lidar coordinate system (denoted as L), and a world coordinate system (denoted as W). The state variables to be optimized include the IMU position in the world coordinate system. attitude speed Gyroscope zero bias b g accelerometer zero bias b a The geometric parameters M of the lane lines W And the geometric parameters N of the road pole W To simplify the description, we can... M W and N W These are abbreviated as P, R, v, M, and N. Among them, P, R, and v can be collectively referred to as position state variables, and M and N can be collectively referred to as attitude state variables.

[0082] In this embodiment of the invention, the pose constraint relationships include global pose constraint relationships, IMU pre-integration constraint relationships, velocity constraint relationships, and point cloud registration constraint relationships. The establishment process of each pose constraint relationship is illustrated below.

[0083] (1) Global pose constraints

[0084] Global pose constraints, also known as absolute pose constraints, are used to estimate the current pose based on the pose at the previous moment and the pose change from the previous moment to the current moment. The pose change is obtained through IMU integration. If the difference between the estimated current pose and the pose observation value of the integrated navigation device is less than a preset difference threshold, then the global pose constraint relationship corresponding to the current moment is established based on the pose state variables and the pose observation value of the integrated navigation device; otherwise, the global pose constraint relationship corresponding to the current moment is discarded.

[0085] Global pose constraints can be represented by the following expression:

[0086]

[0087] Where, r I (R,P) represents the global pose constraint, which can be simply referred to as r. I , This represents the position observation value of the integrated navigation equipment in the world coordinate system. This represents the attitude observation value of the integrated navigation device in the world coordinate system, and log() represents the mapping from Lie group to Lie algebra.

[0088] (2) IMU pre-integration constraint relationship

[0089] The IMU can measure angular velocity and acceleration in real time. By pre-integrating, constraints on the IMU's position, attitude, and velocity can be obtained, increasing the smoothness of state estimation. The IMU pre-integration constraints are expressed by the following expression:

[0090]

[0091] Where, r B (R,P,v,b a ,b g ) represents the IMU pre-integration constraint, which can be simply referred to as r. B t represents time, g represents gravitational acceleration, Δt represents the time difference between time t+1 and time t, and ΔP t(t+1) This represents the change in position between time t and time t+1. Δv represents the reciprocal of the attitude change between time t and time t+1. t(t+1)This represents the change in velocity between time t and time t+1, and log() represents the mapping from a Lie group to a Lie algebra.

[0092] (3) Velocity constraint relationship

[0093] Wheel speed gauges can measure the speed of wheels in a world coordinate system.

[0094]

[0095] in, This represents the observation value from the wheel speed gauge, where v represents the speed state variable. It is a 3×1 vector where the second term is 1 and the rest are 0, n v This indicates the measurement noise of the wheel speed gauge.

[0096] Combining the extrinsic parameters between the IMU coordinate system and the vehicle coordinate system, the velocity in the IMU coordinate system can be obtained as follows:

[0097]

[0098] Where ω represents the angular velocity measured by the IMU gyroscope. The lever arm value from the vehicle coordinate system to the IMU coordinate system can be obtained through external parameter calibration.

[0099] In summary, an error function regarding velocity can be established based on observations from the wheel speedometer and gyroscope:

[0100]

[0101] Where, r V (v) represents the IMU pre-integration constraint, which can be simply referred to as r. V .

[0102] (4) Point cloud registration constraints

[0103] For each keyframe in the preset sliding window (hereinafter referred to as the sliding window), the k-D-Tree algorithm can be used sequentially to establish the association between feature points and the local map. The pose of each keyframe is used as the initial value for the nearest neighbor search in the KD-Tree algorithm. For the current frame, the pose of the previous keyframe and the inter-frame pose change obtained by IMU integration are used to determine the initial value for point cloud registration. Then, based on the nearest neighbor feature point set obtained by the search, corresponding line error functions and surface error functions are established for each line feature point and surface feature point. That is, for line feature points and surface feature points, the KD-Tree algorithm is used to perform nearest neighbor search in the local map based on the initial registration pose to obtain the corresponding nearest neighbor feature point set, and line error functions and surface error functions are established based on the corresponding nearest neighbor feature point set.

[0104] For any linear feature point x l Line fitting can be performed based on the nearest neighbor set. The covariance matrix C of the nearest neighbor set is calculated, yielding the corresponding eigenvalues ​​V and eigenvectors E. If the maximum eigenvalue is significantly larger than the other two, the line fitting is considered successful, and the eigenvector corresponding to the maximum eigenvalue represents the direction of the line. If the line fitting is successful, the line error function can be determined based on the distance from the feature point to the corresponding fitted line. l The distance to the corresponding line, the line error function is expressed by the following expression:

[0105]

[0106] Where, r p2l Representing feature point x l The distance to the corresponding fitted line, where D represents the direction of the fitted line and M represents the mean coordinates of the nearest neighbor feature point set. and This represents the extrinsic parameters from the lidar coordinate system to the IMU coordinate system;

[0107] For any feature point x on the surface p Plane fitting needs to be performed based on the nearest neighbor set. Singular Value Decomposition (SVD) can be used to fit the plane, and the equation of the fitted plane is expressed as n. T x + d = 0, where n represents the normal vector of the plane and d is the corresponding coefficient. If the plane fitting is successful, the surface error function is determined based on the distance from the surface feature points to the corresponding fitted plane. p The distance to the plane, i.e., the surface error function, is expressed by the following expression:

[0108]

[0109] Where, r p2p Represents a planar feature point x p The distance to the corresponding fitted plane, and This represents the extrinsic parameters from the lidar coordinate system to the IMU coordinate system.

[0110] Accordingly, the point cloud registration constraint is represented by the following expression:

[0111]

[0112] Where, r G (R,P) represents the point cloud registration constraint, which can be simply referred to as r G n l n represents the number of line feature points contained in the current keyframe. pThis represents the number of planar feature points contained in the current keyframe, and ∑ is the preset registration information matrix.

[0113] Step 202: Perform semantic segmentation processing on the image acquired by the image acquisition device to obtain the two-dimensional contour of the region where the semantic object is located.

[0114] The semantic objects in this embodiment of the invention may include lane lines and / or road poles. For ease of explanation, the following example includes both lane lines and road poles. First, pixel regions corresponding to lane lines and road poles can be detected from the image using a deep learning-based semantic segmentation method. Then, the 3D information of the lane lines and road poles is calculated by combining the depth information provided by the LiDAR. To facilitate data association and joint optimization of semantic information between frames, the lane lines and road poles can be parameterized using geometric shapes.

[0115] For lane lines, the corresponding three-dimensional contours include spatial rectangles. The representational quantities of a spatial rectangle include the center point (C), normal vector (N), long side vector (T), length (L), and width (W). Each dashed lane line corresponds to one spatial rectangle, and each solid lane line corresponds to multiple spatial rectangles based on its length and curvature.

[0116] For example, semantic segmentation is used to extract pixel regions belonging to lane lines from an image, and then image processing can be used to further obtain the 2D contour of the lane line regions.

[0117] For a road pole, the corresponding three-dimensional profile includes a cylinder. The representative quantities of the cylinder include the center point of the top surface (also known as the starting point, denoted as S), the center point of the bottom surface (also known as the ending point, denoted as E), and the radius (d).

[0118] For example, semantic segmentation is used to extract pixel regions belonging to the pole from the image, and then image processing can be used to further obtain the 2D outline of the pole.

[0119] Step 203: Project the point cloud collected by the point cloud acquisition device onto the image, and determine the target three-dimensional point corresponding to the semantic object based on the two-dimensional contour.

[0120] For example, by using the camera's intrinsic parameters and the extrinsic parameters between the camera and the LiDAR, 3D points measured by the LiDAR can be projected onto an image, and LiDAR points corresponding to lane lines and road poles can be obtained based on the 2D contours.

[0121] Step 204: Perform plane fitting or line fitting on the target 3D points to obtain the corresponding spatial equations.

[0122] For lane lines, since the corresponding area is usually on a plane, the corresponding spatial plane equation can be obtained by performing plane fitting on the above-mentioned LiDAR points.

[0123] The equation of the spatial plane is expressed by the following expression:

[0124]

[0125] Where n = (n1 n2 n3) represents the unit normal vector, and d represents the distance from the origin of the coordinate system of the image acquisition device to the spatial plane.

[0126] For a road pole, by fitting a straight line to the laser points belonging to the road pole, the equation of the straight line corresponding to the road pole can be obtained, and the distance from each laser point belonging to the road pole to the straight line can be calculated. The average of the distances can be used as the radius d of the road pole, and the straight line can be represented by the direction vector m = (m1 m2 m3) and a point p on the road pole = (p1 p2 p3).

[0127] Step 205: Determine the three-dimensional coordinates of the target three-dimensional point based on the pixel coordinates of the two-dimensional contour, the intrinsic parameters of the image acquisition device, and the extrinsic parameters between the image acquisition device and the point cloud acquisition device.

[0128] For lane lines, based on geometric relationships, the coordinates of the 3D points corresponding to the 2D contours, i.e., the 3D coordinates of the target 3D points, can be represented by the following expression:

[0129]

[0130] Where u and v represent the pixel coordinates of points on the two-dimensional contour, f x f y c x and c y H represents the camera intrinsic parameters, H = n1f y (uc x )+n2f x (vc y )+n3f x f y .

[0131] For a road pole, since the pixel coordinates of the two endpoints and the equation of the line are known, the three-dimensional coordinates of the two endpoints S and E can be obtained according to geometric relationships. That is, the three-dimensional coordinates of the target three-dimensional point are expressed by the following expression:

[0132]

[0133] Where u and v represent the pixel coordinates of the two endpoints, f x f y c x and c y This indicates the camera's internal parameters.

[0134] Step 206: Fit the target 3D point to the spatial equation and the 3D coordinates of the target 3D point to obtain the observed value of the 3D contour corresponding to the semantic object.

[0135] For lane markings, PCA can be used to fit the 3D point set corresponding to the lane marking profile, yielding the center point, normal vector, long side vector, length, and width of the 3D rectangle. Similarly, for road poles, PCA can be used to fit the 3D point set corresponding to the road pole, yielding the center point of the top surface, center point of the bottom surface, and radius of the pole.

[0136] Step 207: For each keyframe in the preset sliding window, establish the association between the observed values ​​of the 3D contours in the current keyframe and the set of semantic object state variables, so as to update the semantic object state variables in the set of semantic object state variables, and establish the error function corresponding to each 3D contour in the current keyframe.

[0137] For lane lines, to improve mapping accuracy by merging all lane line observations within the sliding window, a dynamically updated set of lane line state variables M can be maintained. The lane line rectangle fitted in each frame within the sliding window is associated with M, and an error function is established.

[0138] For a dashed rectangle, the association between the observation and M can be established based on the coordinates of the center point. First, the coordinates of the center point are transformed according to the global pose of the current frame to obtain the coordinates of the center point of the rectangle in the world coordinate system. If a corresponding state variable can be found within a certain threshold range, the association is considered successful and an error function is established. If a corresponding state variable cannot be found, a new state variable is created and an error function is established.

[0139] The error function for dashed lane lines is expressed by the following formula:

[0140]

[0141] Where, r d (R,P,M) represents the error function for dashed lane lines, which can be abbreviated as r. d P represents the position of the current keyframe in the world coordinate system, R represents the attitude of the current keyframe in the world coordinate system, M represents the spatial rectangle state variable, C represents the center point, N represents the normal vector, T represents the long side vector, W represents the length, and W represents the width; the superscript W represents the world coordinate system, corresponding to the state variable; the superscript I represents the inertial navigation coordinate system, corresponding to the observation value.

[0142] For solid-line spatial rectangles, the association method is similar to that for dashed-line spatial rectangles. That is, the association between the observation and M can be established based on the coordinates of the center point. First, the coordinates of the center point are transformed according to the global pose of the current frame to obtain the coordinates of the center point of the rectangle in the world coordinate system. If a corresponding state variable can be found within a certain threshold range, the association is considered successful, and an error function is established. If a corresponding state variable cannot be found, a new state variable is created, and an error function is established.

[0143] Since the solid line is continuous and there is no strict correspondence between the state variable and the fitting results of each frame, it is necessary to establish the distance from the center point of the state variable to the center line of each observation. Furthermore, there is no need to optimize the length of the rectangle. The error function for the solid line lane is expressed by the following expression:

[0144]

[0145] Where, r s (R,P,M) represents the error function for solid lane markings, which can be abbreviated as r. s P represents the position of the current keyframe in the world coordinate system, R represents the attitude of the current keyframe in the world coordinate system, M represents the spatial rectangle state variable, C represents the center point, N represents the normal vector, T represents the long side vector, and W represents the width; the superscript W represents the world coordinate system, corresponding to the state variable; the superscript I represents the inertial navigation coordinate system, corresponding to the observation value.

[0146] By combining the lane line fitting results of all lane lines in a frame, the lane line error function corresponding to the spatial rectangle in the current keyframe can be obtained, which can be expressed by the following expression:

[0147]

[0148] Where, r L (R,P,M) represents the lane line error function corresponding to the spatial rectangle in the current keyframe, and can be abbreviated as r. L n d n represents the number of space rectangle state variables corresponding to the dashed lane lines in the current keyframe. s ∑ represents the number of spatial rectangle state variables corresponding to the solid lane lines in the current keyframe. d Represents the preset dashed information matrix, ∑ s This represents a preset solid line information matrix.

[0149] For poles, to improve mapping accuracy by integrating all pole observations within the sliding window, a dynamically updated set of pole state variables, N, can be maintained. Each pole fitted within the sliding window is associated with N, and an error function is established. The association between the observation and N can be established based on the coordinates of the pole's two endpoints. First, the endpoint coordinates are transformed according to the global pose of the current frame to obtain the coordinates of the endpoints in the world coordinate system. For the set of state variables N, if the distance difference between a certain state variable and the current pole's endpoints is less than a threshold, the association is considered successful, and an error function is established. If no corresponding state variable can be found, a new state variable is created, and an error function is established.

[0150] The pole error function is expressed by the following expression:

[0151]

[0152] Where, r p (R,P,N) represents the error function for dashed lane markings, which can be abbreviated as r. p P represents the position of the current keyframe in the world coordinate system, R represents the pose of the current keyframe in the world coordinate system, S represents the coordinates of the center point of the top surface of the cylinder, E represents the coordinates of the center point of the bottom surface of the cylinder, and d represents the radius of the cylinder.

[0153] By combining the fitting results of all road poles in a frame, the road pole error function corresponding to the pole in the current keyframe can be obtained, which can be expressed by the following expression:

[0154]

[0155] Where, r P (R,P,N) represents the lane line error function corresponding to the spatial rectangle in the current keyframe, and can be abbreviated as r. P n p This represents the number of road poles included in the road pole fitting result in the current keyframe, that is, the number of column state variables corresponding to the road poles, Σ. p This is a preset pole information matrix.

[0156] Step 208: Establish semantic constraint relationships based on the error functions corresponding to each keyframe in the preset sliding window.

[0157] For example, semantic constraint relations can be represented as:

[0158]

[0159] Where, n k This indicates the number of keyframes in the preset sliding window.

[0160] Step 209: Construct a joint optimization objective function for each keyframe within a preset sliding window based on pose constraints and semantic constraints.

[0161] The pre-integration constraints r of each frame within the combined sliding window B Speed ​​constraint r V Absolute pose constraints r I Point cloud registration constraints r G Lane line constraints r L and pole constraint r P The following joint optimization objective function can be established:

[0162]

[0163] Where X = {R, P, v, b} a ,b g ,M},n k This represents the number of keyframes in the preset sliding window, and ∑ represents the preset information matrix.

[0164] Step 210: Solve the joint optimization objective function to obtain the estimated values ​​of each state variable. The estimated values ​​of the semantic object state variables are denoted as semantic results, and the estimated values ​​of the position state variables and attitude state variables are denoted as pose results.

[0165] For example, the Ceres-Solver algorithm can be used to solve the joint optimization objective function to obtain the estimated values ​​of each state variable.

[0166] Step 211: Construct a semantic map and a point cloud map based on the semantic results and pose results.

[0167] For example, for lane lines, the four corner points of a spatial rectangle can be calculated based on geometric relationships and semantic results. Connecting these corner points yields the 3D contour of the lane line. Combining all the 3D contours of the lane lines creates a semantic map of the lane lines. Specifically, for dashed lane lines, the 3D contour corresponds to the four corner points of the spatial rectangle connected sequentially. For solid lane lines, the 3D contour corresponds to the two side lines obtained by connecting the corner points along the length direction in spatial order.

[0168] For example, for a road pole, the center point of the top surface and the center point of the bottom surface can be determined based on the geometric relationship and semantic results. Then, the 3D outline of the road pole can be constructed based on the radius. By combining the 3D outlines of all road poles, a semantic map of the road pole can be obtained.

[0169] For point cloud maps, the motion-compensated point clouds can be stacked using optimized poses to obtain a dense point cloud map.

[0170] By adopting the above-mentioned technical solution, this invention first establishes global and local consistency constraints on the system state by fusing and modeling observation data from integrated navigation devices, wheel speedometers, cameras, and lidar. This effectively improves the accuracy and robustness of pose estimation and enables normal mapping in scenarios such as tunnels and cross-sea bridges. Through image-based semantic segmentation technology, pixel regions belonging to lane lines and road poles are effectively detected. Then, combined with 3D observations from lidar, 3D information of lane lines and road poles is calculated. Finally, temporal information is used to jointly optimize lane line and road pole observations at different times, thereby improving the accuracy and consistency of the semantic map.

[0171] To verify the effectiveness of the technical solution in the embodiments of the present invention, a data acquisition vehicle equipped with sensors such as GNSS, IMU, wheel speedometer, camera, and lidar was used to collect road data under various complex road conditions for experimental verification. Figure 4 This is a schematic diagram of the map construction result of the bridge scene provided in the embodiment of the present invention. Sub-figure A is the bridge scene map, sub-figure B is the constructed point cloud map, sub-figure C is a schematic diagram of the constructed semantic map, and sub-figure D is a curve of the difference between the optimized position (X, Y and Z represent the three coordinate components in the spatial coordinate system) and the true value. Figure 5 This is a schematic diagram of the map construction result of a tunnel scene provided in an embodiment of the present invention. Sub-figure A is the tunnel scene map, sub-figure B is the constructed point cloud map, sub-figure C is a schematic diagram of the constructed semantic map, and sub-figure D is a curve showing the difference between the optimized position (X, Y, and Z represent the three coordinate components in the spatial coordinate system) and the true value. Data was collected by a data acquisition vehicle. Figure 4 Relevant data and collection in the bridge scene shown Figure 5 The relevant data in the tunnel scenario shown were used to construct a map using the technical solution provided in this embodiment of the invention, taking lane lines as an example of semantic objects. The data collection vehicle was equipped with a high-precision commercial integrated navigation device, and its post-processing results could be used as the true values ​​of pose. To verify the accuracy of the lane line map, the spatial coordinates of some lane line dashed corner points (hereinafter referred to as absolute position points) were measured using surveying equipment. See also... Figure 4 and Figure 5 It can be seen that the technical solution provided by the embodiments of the present invention can still build maps under complex road conditions such as cross-sea bridges and tunnels, and the error between the calculated position and the true value is within 10cm. In addition, the difference between the corner coordinates of the constructed lane line map and the absolute position point is also within 10cm, which effectively ensures the accuracy and precision of the mapping.

[0172] Figure 6This is a structural block diagram of a map building device provided in an embodiment of the present invention. The device can be implemented by software and / or hardware, and is generally integrated into a computer device. It can build maps by executing map building methods. Figure 6 As shown, the device includes:

[0173] The data acquisition module 601 is used to acquire sensor data collected by a preset sensor;

[0174] The constraint relationship establishment module 602 is used to establish the pose constraint relationship of the movable object and the semantic constraint relationship of the semantic object based on sensor data;

[0175] The joint solution module 603 is used to perform joint optimization solution based on the pose constraint relationship and the semantic constraint relationship to determine the semantic result of the semantic object and the pose result of the movable object;

[0176] The map building module 604 is used to build a semantic map and a point cloud map based on the semantic results and the pose results.

[0177] The map building device provided in this embodiment of the invention integrates the construction process of semantic map and point cloud map, and uses pose constraint relationship and semantic constraint relationship for joint solution, so that the semantic result and pose result can form a mutual constraint relationship, which can quickly and accurately obtain semantic map and point cloud map, and thus obtain a more accurate high-precision map, which is beneficial to improving the autonomous movement performance of movable objects.

[0178] Optionally, the sensor data collected by the preset sensor includes images collected by the image acquisition device and point clouds collected by the point cloud acquisition device. The constraint relationship establishment module is specifically used to: determine the observation values ​​of the three-dimensional contours corresponding to each semantic object based on the images and the point clouds; and establish semantic constraint relationships based on the observation values ​​of the three-dimensional contours.

[0179] Optionally, determining the observed value of the three-dimensional contour corresponding to the semantic object based on the image and the point cloud includes: performing semantic segmentation processing on the image to obtain a two-dimensional contour of the region where the semantic object is located; projecting the point cloud onto the image and determining the target three-dimensional point corresponding to the semantic object based on the two-dimensional contour; and fitting the target three-dimensional point to obtain the observed value of the three-dimensional contour corresponding to the semantic object.

[0180] Optionally, fitting the target 3D point to obtain the observed value of the 3D contour corresponding to the semantic object includes: performing planar fitting or linear fitting on the target 3D point to obtain the corresponding spatial equation; determining the 3D coordinates of the target 3D point based on the pixel coordinates of the 2D contour, the intrinsic parameters of the image acquisition device, and the extrinsic parameters between the image acquisition device and the point cloud acquisition device; and fitting the target 3D point based on the spatial equation and the 3D coordinates of the target 3D point to obtain the observed value of the 3D contour corresponding to the semantic object.

[0181] Optionally, the semantic object includes lane lines, the three-dimensional contour corresponding to the lane lines includes spatial rectangles, and the representational quantities of the spatial rectangles include at least one of center point, normal vector, long side vector, length and width. The fitting method for the target three-dimensional points corresponding to the lane lines is planar fitting, thereby obtaining the corresponding spatial plane equation; wherein, each dashed lane line corresponds to one spatial rectangle, and each solid lane line corresponds to multiple spatial rectangles according to its length and curvature.

[0182] Optionally, the semantic object includes a road pole, and the three-dimensional contour corresponding to the road pole includes a cylinder. The representative quantity of the cylinder includes at least one of the center point of the top surface, the center point of the bottom surface, and the radius. The fitting method for the target three-dimensional point corresponding to the road pole is linear fitting, thereby obtaining the corresponding spatial linear equation.

[0183] Optionally, establishing semantic constraint relationships based on the observations of the three-dimensional contours includes: for each keyframe in a preset sliding window, establishing a correlation between the observations of the three-dimensional contours in the current keyframe and the set of semantic object state variables, so as to update the semantic object state variables in the set of semantic object state variables, and establishing an error function corresponding to each three-dimensional contour in the current keyframe; and establishing semantic constraint relationships according to the error functions corresponding to each keyframe in the preset sliding window.

[0184] Optionally, for lane lines, establishing the association between the observed values ​​of the three-dimensional contour in the current keyframe and the set of semantic object state variables to update the semantic object state variables in the set of semantic object state variables includes: for each spatial rectangle in the current keyframe, calculating the first distance between the observed value of the center point of the current spatial rectangle and the center point of each rectangle state variable in the set of lane line state variables in the same coordinate system; if there is a first distance less than a first preset distance threshold, then the current spatial rectangle is associated with the corresponding rectangle state variable; if not, then the rectangle state variable corresponding to the current spatial rectangle is created in the set of lane line state variables.

[0185] Optionally, the error function in the current keyframe includes the lane line error function corresponding to the spatial rectangle and / or the road pole error function corresponding to the column. The lane line error function includes the error function of each dashed lane line and the error function of each solid lane line.

[0186] Optionally, the error function for the dashed lane line includes: error functions of the state variables with respect to the center point, normal vector, long side vector, length, and width, and error functions of the state variables of the normal vector and long side vector with respect to the unit vector, respectively.

[0187] Optionally, the error function for the solid lane line includes: error functions of the state variables with respect to the normal vector, the long side vector, and the width with respect to the observed values; the second distance of the state variable of the center point to the observed center line; and error functions of the state variables of the normal vector and the long side vector with respect to the unit vector, respectively.

[0188] Optionally, the preset sensor includes one or more of GNSS, inertial measurement unit (IMU), wheel speedometer, image acquisition device, and point cloud acquisition device; wherein the IMU includes an accelerometer and a gyroscope, and the GNSS can also be combined with the IMU to form a combined navigation device; wherein the pose constraint relationship includes one or more of the following: global pose constraint relationship, IMU pre-integration constraint relationship, velocity constraint relationship, and point cloud registration constraint relationship.

[0189] Optionally, the constraint relationship establishment module is specifically used to: when the pose constraint relationship includes a global pose constraint relationship, estimate the current pose at the current moment based on the pose at the previous moment and the pose change from the previous moment to the current moment, wherein the pose change is obtained through IMU integration; if the difference between the estimated current pose and the pose observation value is less than a preset difference threshold, then establish the global pose constraint relationship corresponding to the current moment based on the pose state variables and the pose observation value; otherwise, discard the global pose constraint relationship corresponding to the current moment.

[0190] Optionally, the joint solution module is specifically used to: construct a joint optimization objective function corresponding to each keyframe within a preset sliding window based on the pose constraint relationship and the semantic constraint relationship; solve the joint optimization objective function to obtain the estimated values ​​of each state variable; wherein, the estimated values ​​of the semantic object state variables are recorded as semantic results, and the estimated values ​​of the position state variables and the pose state variables are recorded as pose results.

[0191] Optionally, the map construction module is specifically used for: calculating the corner points of the semantic object based on the semantic results and geometric relationships; connecting each corner point to obtain the 3D contour of the semantic object; combining the 3D contours of each semantic object to obtain a semantic map of the semantic object; and overlaying the point cloud based on the pose results to obtain a point cloud map; wherein, for dashed lane lines, the 3D contour corresponds to the four corner points of the spatial rectangle connected sequentially, and for solid lane lines, the 3D contour corresponds to the two side lines obtained by connecting the corner points in the length direction in spatial order.

[0192] Optionally, the point cloud registration constraints are obtained as follows: For the point cloud of each key frame within a preset sliding window, each point is divided into line feature points and planar feature points based on the curvature information of the points in the point cloud; for each key frame within the preset sliding window, the pose of the previous key frame is superimposed with the inter-frame pose change obtained by IMU integration to obtain the initial registration pose corresponding to the current key frame; for line feature points and planar feature points, nearest neighbor search is performed in the local map based on the initial registration pose to obtain the corresponding nearest neighbor feature point set, wherein the local map includes the feature points of each key frame in the preset sliding window; line error function and planar error function are established based on the corresponding nearest neighbor feature point set; and point cloud registration constraints are determined based on the line error function and planar error function within the preset sliding window.

[0193] Optionally, a line error function is established based on the nearest neighbor feature point set corresponding to the line feature point, including: performing line fitting based on the nearest neighbor feature point set corresponding to the line feature point, and determining the line error function based on the distance from the line feature point to the corresponding fitted line. A surface error function is established based on the nearest neighbor feature point set corresponding to the plane feature point, including: performing plane fitting based on the nearest neighbor feature point set corresponding to the plane feature point, and determining the surface error function based on the distance from the plane feature point to the corresponding fitted plane.

[0194] This invention provides a computer device that can integrate the map building apparatus provided in this invention. Figure 7 This is a structural block diagram of a computer device provided in an embodiment of the present invention. The computer device 700 may include: a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor. When the processor 702 executes the computer program, it implements the map construction method as described in this embodiment of the present invention. The computer device can be integrated into a movable object; in this case, the computer device can also be considered the movable object itself, such as a vehicle. The computer device can also be externally connected to the movable object or communicate with the movable object.

[0195] The computer device provided in this embodiment of the invention integrates the construction process of semantic map and point cloud map, and uses pose constraint relationship and semantic constraint relationship for joint solution, so that the semantic result and pose result can form a mutual constraint relationship, which can quickly and accurately obtain semantic map and point cloud map, and thus obtain a more accurate high-precision map, which is beneficial to improving the autonomous movement performance of movable objects.

[0196] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, implement the map construction method provided in this invention.

[0197] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDRRAM, SRAM, EDORAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0198] The map building apparatus, device, and storage medium provided in the above embodiments can execute the map building method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the above embodiments can be found in the map building method provided in any embodiment of the present invention.

[0199] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A map construction method, characterized in that, include: Acquire sensor data collected by preset sensors; Based on the sensor data, establish the pose constraint relationship of the movable object and the semantic constraint relationship of the semantic object; The semantic result of the semantic object and the pose result of the movable object are determined by joint optimization based on the pose constraint relationship and the semantic constraint relationship. as well as Construct a semantic map and a point cloud map based on the semantic results and the pose results; The pose constraint relationship includes one or more of the following: global pose constraint relationship, IMU pre-integration constraint relationship, velocity constraint relationship and point cloud registration constraint relationship; The step of jointly optimizing and solving the problem based on the pose constraints and the semantic constraints to determine the semantic result and the pose result of the movable object includes: Based on the pose constraint relationship and the semantic constraint relationship, construct a joint optimization objective function corresponding to each key frame within a preset sliding window; The joint optimization objective function is solved to obtain the estimated values ​​of each state variable; wherein, the estimated value of the semantic object state variable is recorded as the semantic result, and the estimated values ​​of the position state variable and the attitude state variable are recorded as the pose result.

2. The method according to claim 1, characterized in that, The sensor data includes images acquired by an image acquisition device and point clouds acquired by a point cloud acquisition device. Based on the sensor data, semantic constraint relationships for semantic objects are established, including: Based on the image and the point cloud, determine the observation values ​​of the three-dimensional contours corresponding to each semantic object; Semantic constraint relationships are established based on the observations of the three-dimensional contour.

3. The method according to claim 2, characterized in that, The step of determining the observation value of the three-dimensional contour corresponding to the semantic object based on the image and the point cloud includes: The image is subjected to semantic segmentation to obtain the two-dimensional contour of the region where the semantic object is located; The point cloud is projected onto the image, and the target three-dimensional point corresponding to the semantic object is determined based on the two-dimensional contour. The target 3D points are fitted to obtain the observed values ​​of the 3D contour corresponding to the semantic object.

4. The method according to claim 3, characterized in that, The process of fitting the target 3D points to obtain the observed values ​​of the 3D contour corresponding to the semantic object includes: The target three-dimensional points are fitted with a plane or a line to obtain the corresponding spatial equations. The three-dimensional coordinates of the target three-dimensional point are determined based on the pixel coordinates of the two-dimensional contour, the intrinsic parameters of the image acquisition device, and the extrinsic parameters between the image acquisition device and the point cloud acquisition device. The target 3D point is fitted based on the spatial equation and the 3D coordinates of the target 3D point to obtain the observed value of the 3D contour corresponding to the semantic object.

5. The method according to claim 4, characterized in that, The semantic object includes lane lines, and the three-dimensional contour corresponding to the lane lines includes spatial rectangles. The representation of the spatial rectangle includes at least one of the following: center point, normal vector, long side vector, length, and width. The fitting method for the target three-dimensional points corresponding to the lane lines is planar fitting, thereby obtaining the corresponding spatial plane equation. Each dashed lane line corresponds to one spatial rectangle, and each solid lane line corresponds to multiple spatial rectangles according to its length and curvature.

6. The method according to claim 5, characterized in that, The semantic object includes a road pole, and the three-dimensional contour corresponding to the road pole includes a cylinder. The representative quantity of the cylinder includes at least one of the center point of the top surface, the center point of the bottom surface, and the radius. The fitting method for the target three-dimensional point corresponding to the road pole is linear fitting, thereby obtaining the corresponding spatial linear equation.

7. The method according to claim 6, characterized in that, The establishment of semantic constraint relationships based on the observations of the three-dimensional contour includes: For each keyframe in the preset sliding window, establish the association between the observed values ​​of the 3D contours in the current keyframe and the set of semantic object state variables, so as to update the semantic object state variables in the set of semantic object state variables, and establish the error function corresponding to each 3D contour in the current keyframe. Semantic constraint relationships are established based on the error functions corresponding to each keyframe in the preset sliding window.

8. The method according to claim 7, characterized in that, For lane lines, establishing the association between the observed values ​​of the 3D contour in the current keyframe and the set of semantic object state variables, in order to update the semantic object state variables in the set of semantic object state variables, includes: For each spatial rectangle in the current keyframe, calculate the first distance between the observed value of the center point of the current spatial rectangle and the center point of each rectangle state variable in the lane line state variable set in the same coordinate system. If there is a first distance less than the first preset distance threshold, then associate the current spatial rectangle with the corresponding rectangle state variable. If not, create the rectangle state variable corresponding to the current spatial rectangle in the lane line state variable set.

9. The method according to claim 7, characterized in that, The error functions in the current keyframe include the lane line error function corresponding to the spatial rectangle and / or the road pole error function corresponding to the column. The lane line error function includes the error function of each dashed lane line and the error function of each solid lane line.

10. The method according to claim 7, characterized in that, The error function for dashed lane markings includes: Error functions relating the state variables of the center point, normal vector, long side vector, length, and width to the observed values, and error functions relating the state variables of the normal vector and long side vector to the unit vector, respectively.

11. The method according to claim 7, characterized in that, The error function for solid lane markings includes: Error functions relating the state variables of the normal vector, long side vector, and width to the observed values; the second distance from the state variable of the center point to the observation centerline; and the error functions relating the state variables of the normal vector and long side vector to the unit vector, respectively.

12. The method according to any one of claims 1-11, characterized in that, The preset sensors include one or more of the following: Global Navigation Satellite System (GNSS), Inertial Measurement Unit (IMU), Wheel Speed ​​Meter, Image Acquisition Device, and Point Cloud Acquisition Device; wherein the IMU includes an accelerometer and a gyroscope.

13. The method according to claim 12, characterized in that, When the pose constraint relationship includes a global pose constraint relationship, establishing the global pose constraint relationship of the movable object based on the collected sensor data includes: Based on the pose of the previous moment and the pose change from the previous moment to the current moment, estimate the current pose at the current moment, where the pose change is obtained by IMU integration. If the difference between the estimated current pose and the pose observation is less than a preset difference threshold, then the global pose constraint relationship corresponding to the current moment is established based on the pose state variables and the pose observation; otherwise, the global pose constraint relationship corresponding to the current moment is discarded.

14. The method according to claim 1, characterized in that, Constructing a semantic map and a point cloud map based on the semantic results and the pose results includes: Calculate the corner points of the semantic object based on the semantic results and geometric relationships; By connecting the corner points, the 3D outline of the semantic object is obtained; By combining the 3D outlines of each semantic object, a semantic map of the semantic objects is obtained; The point cloud is overlaid based on the pose results to obtain a point cloud map; For dashed lane lines, the 3D outline corresponds to the four corner points of the spatial rectangle connected sequentially. For solid lane lines, the 3D outline corresponds to the two side lines obtained by connecting the corner points in the length direction in spatial order.

15. The method according to claim 1, characterized in that, The point cloud registration constraints are obtained in the following way: For the point cloud of each key frame within the preset sliding window, each point is divided into line feature points and planar feature points based on the curvature information of the points in the point cloud. For each keyframe within the preset sliding window, the pose of the previous keyframe is superimposed with the inter-frame pose change obtained by IMU integration to obtain the initial registration pose corresponding to the current keyframe. For line feature points and planar feature points, nearest neighbor search is performed in the local map according to the initial registration pose to obtain the corresponding nearest neighbor feature point set. The local map includes the feature points of each key frame in the preset sliding window. Establish line error function and surface error function respectively based on the corresponding nearest neighbor feature point set; The point cloud registration constraints are determined based on the line error function and surface error function within the preset sliding window.

16. The method according to claim 15, characterized in that, The line error function is established based on the set of nearest neighbor feature points corresponding to the line feature points, including: Line fitting is performed based on the set of nearest neighbor feature points corresponding to the line feature points, and the line error function is determined based on the distance from the line feature points to the corresponding fitted lines. A surface error function is established based on the set of nearest neighbor feature points corresponding to the planar feature points, including: Plane fitting is performed based on the nearest neighbor feature point set corresponding to the planar feature point, and the surface error function is determined based on the distance from the planar feature point to the corresponding fitting plane.

17. A map building device, characterized in that, include: The data acquisition module is used to acquire sensor data collected by preset sensors; The constraint relationship establishment module is used to establish the pose constraint relationship of the movable object and the semantic constraint relationship of the semantic object based on the sensor data. The joint solution module is used to perform joint optimization solution based on the pose constraint relationship and the semantic constraint relationship to determine the semantic result of the semantic object and the pose result of the movable object; A map building module is used to build a semantic map and a point cloud map based on the semantic results and the pose results. The pose constraint relationship includes one or more of the following: global pose constraint relationship, IMU pre-integration constraint relationship, velocity constraint relationship and point cloud registration constraint relationship; The joint solver module is further used for: Based on the pose constraint relationship and the semantic constraint relationship, construct a joint optimization objective function corresponding to each key frame within a preset sliding window; The joint optimization objective function is solved to obtain the estimated values ​​of each state variable; wherein, the estimated value of the semantic object state variable is recorded as the semantic result, and the estimated values ​​of the position state variable and the attitude state variable are recorded as the pose result.

18. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-16.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-16.

Images