Automatic driving lightweight semantic map construction and updating method based on low earth orbit satellite prior information and perception uncertainty

Abstract

The invention discloses an automatic driving lightweight semantic map construction and updating method based on low earth orbit satellite prior information and perception uncertainty, and for high-precision semantic map construction, the method comprises the steps of S1, image semantic segmentation, S2, inverse perspective change, S3, pose map optimization and S4, single vehicle mapping. And S5, map merging and making. The updating of the high-precision semantic map comprises the steps of S1, real-time sensing and positioning of the bicycle and S2, updating judgment of the map. The method mainly solves the problem of building and updating the urban road automatic driving high-precision semantic map. In map construction, a low-orbit satellite is used as a track signal, and meanwhile, a RTK signal and an IMU odometer are subjected to mapping optimization to perform more accurate track recovery. In the map updating process, for the problem that the perception of a camera sensor is uncertain, a real-time positioning error is used as a basis for updating the map, a low-orbit satellite signal is used as a positioning truth value, and whether the map is updated or not is judged by using the game theory, so that the problem of wrong updating caused by the uncertainty of the perception of the camera is weakened.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and specifically to a lightweight semantic map construction and update method for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty. Background Art

[0002] With the continuous progress of technology, the popularity and coverage of autonomous driving systems have developed rapidly. Autonomous driving can not only improve urban traffic conditions, effectively reduce traffic congestion and traffic accidents, but also a full range of autonomous driving can serve commercial fields such as logistics transportation and public transportation. In autonomous driving tasks, high-precision maps play an important role. It uses semantic information to provide accurate environmental data, which is widely used in downstream tasks such as perception, positioning, planning, and control. High-precision maps can ensure the accurate position and reliable navigation performance of the vehicle itself. Traditionally, high-precision maps are created offline using methods based on SLAM (Simultaneous Localization and Mapping). However, this method has significant drawbacks. Visual SLAM methods with cameras as the main sensors have large cumulative errors, and the high-precision maps produced have large errors and cannot be used in autonomous driving tasks. The laser SLAM method with lidar as the main sensor, as the most commonly used mapping method, although the accuracy is high enough, the high cost of high-line beam lidar hinders its development. Moreover, for point cloud maps, their huge memory occupancy further increases the production cost. When the environmental data changes, the cost of updating the map cannot be ignored.

[0003] Recently, the construction and update of high-precision maps through crowdsourcing data have attracted people's attention. The crowdsourcing vehicles upload the collected data to the cloud for computing, which can effectively reduce the storage cost and computing cost, and at the same time ensure the timeliness of the map. Usually, the crowdsourcing devices include low-cost cameras, wheel speedometers, IMUs, and Global Navigation Satellite Systems (GNSS). The camera sensor is responsible for obtaining environmental information data, and other sensors are responsible for providing accurate position information. The crowdsourcing devices are installed on vehicles that often repeat the same route. The construction of high-precision maps from crowdsourcing data usually includes two processes: production and update. Therefore, a large amount of real-time map environment data can be obtained at low cost to ensure the high timeliness of the map. However, there are also several disadvantages in crowdsourcing data: 1) In map production, due to the urban canyon effect, GNSS cannot always ensure a reliable attitude. GNSS can only provide centimeter-level accuracy in open areas. In urban scenarios, its signal is easily blocked by high-rise buildings. Inaccurate attitude will seriously affect the map accuracy. 2) In map update, the perception data of a single vehicle has high uncertainty. Appropriately updating the high-definition map by reflecting environmental changes is often unreliable. Various uncertainties brought by low-quality observations and uncertain observations will cause problems such as false updates and slow updates, which will seriously affect the map accuracy. The emergence and rapid popularization of low-earth orbit satellites provide a new idea for crowdsourcing high-precision maps. Low-earth orbit satellites can greatly improve the urban canyon effect. For intelligent vehicles driving in urban environments, low-earth orbit satellites can continuously provide positioning information with an accuracy of about 3 centimeters, which provides extremely robust prior information for the mapping task. The present invention proposes a method for constructing and updating a high-precision semantic map based on the accurate prior positioning information provided by low-earth orbit satellites and real-time perception uncertainty. In the map production stage, the prior positioning value provided by low-earth orbit satellites is used to generate an accurate pose for the map and thus provide an accurate trajectory. Since low-earth orbit satellites can effectively alleviate the canyon effect, the method of the present invention can be used in all urban areas. In the map update stage, based on perception uncertainty, the present invention does not use traditional perception methods, but innovatively uses the real-time positioning information of the vehicle as the judgment basis for map update. Summary of the Invention

[0004] The present invention provides a method for constructing and updating a lightweight semantic map for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty. By using the high-precision positioning prior information provided by low-earth orbit satellites to optimize the map trajectory pose to produce a high-precision map, in the map update part, the traditional perception scheme is discarded to avoid false update problems caused by perception uncertainty. The present invention first introduces real-time positioning, and then compares the real-time positioning information with the prior positioning information of low-earth orbit satellites to determine whether to update the map. To achieve the above object, the implementation steps of the present invention are as follows:

[0005] First, produce a high-precision map, and the specific implementation steps are as follows:

[0006] 1. Image semantic segmentation

[0007] The present invention uses a method based on the traditional convolutional neural network (CNN) for monocular scene semantic segmentation. Specifically, the DeepLabV3+ model is used as the semantic segmentation model. The present invention defines four categories, namely the ground, lane lines, stop lines, and road markings, which also serve as elements of the semantic map.

[0008] 2. Inverse perspective transformation

[0009] After segmentation, the semantic pixels need to be inverse projected from the image plane to the ground plane in the vehicle coordinate system. This process is called inverse perspective mapping. The present invention calibrates the internal parameters of the camera and the external parameters from the camera to the vehicle center coordinate origin offline. Since there is a mapping error in inverse perspective mapping and the error increases with the increase of the scene distance, the present invention artificially sets the projection mapping distance. According to a large number of experiments, the present invention finally determines that the mapping range is a 12m×6m rectangular area. Experiments have proved that if the area is exceeded, the mapping error will increase instantaneously. The present invention assumes that urban roads are all flat. After semantic segmentation, the image pixel coordinates in the image coordinate system are (u, v). Assuming that the image pixel coordinates are projected onto the ground to obtain the coordinates (x u , y v ), then:

[0010]

[0011] where π c (.) is the camera undistortion and projection transformation, and π c (.) -1 represents inverse projection, that is, lifting the image pixels to the world coordinate system space. [R c t c is the external parameter matrix of each image pixel relative to the vehicle center. R c is the rotation matrix of the image pixel relative to the vehicle center, and t c is the translation matrix of the image pixel relative to the vehicle center. λ is a scalar. [] col:1，2，4 means taking the 1st, 2nd, and 4th columns of the matrix.

[0012] 3. Pose graph optimization

[0013] The accurate pose of the vehicle is crucial for the map. GNSS can only provide centimeter-level positioning in open areas and is easily blocked by high-rise buildings in urban areas. Although the inertial measurement unit (IMU) and the wheel speedometer can continuously provide odometry, there is a problem of cumulative drift in the long-term operation of the odometry. To alleviate the drift, the present invention performs pose graph optimization. The present invention abandons the GNSS signal scheme and innovatively introduces the low-earth orbit satellite signal as an optimization factor. The low-earth orbit satellite signal can easily penetrate the blockage of high-rise buildings and can provide accurate pose information in the entire domain. Let the state of the vehicle at a certain moment be s, which includes the position p and the orientation q represented by a quaternion. The quaternion is converted into a rotation matrix using the operation R(q). For pose graph optimization, there are two types of edges. One is the low-earth orbit satellite signal, and the other is the odometry signal. These two types of edges exist at any time. The pose graph optimization can be expressed as:

[0014]

[0015] where s represents the vehicle pose state (position and orientation), and the subscript i represents the i-th moment. r o is the residual factor of the odometry signal. is the odometry measurement value, which includes the position increment and the orientation increment r g is the residual factor of the low-earth orbit satellite. is the low-earth orbit satellite measurement value, that is, the position in the world coordinate system σ represents the weight, which can adopt the standard deviation value.

[0016] It is further defined as follows:

[0017]

[0018] where [.] xyz represents taking the first three elements of the quaternion, approximately equal to an error perturbation on the manifold; p i represents the vehicle position at the i-th moment, q i represents the vehicle orientation at the i-th moment, and R(q i-1 ) is the rotation matrix corresponding to the vehicle orientation at the (i - 1)-th moment.

[0019] 4. Single vehicle mapping

[0020] The pose graph optimization can provide a reliable vehicle pose at each moment. Each frame of semantic image i is converted from the vehicle coordinate system to the global coordinate system according to the pose after the graph optimization. That is:

[0021]

[0022] x w 、y w, z w represent the coordinates of the global coordinate system.

[0023] Through image segmentation, each point is assigned a class label, such as ground, lane line, road sign, and crosswalk. The position of each point in the world coordinate system corresponds to a small area, which may be observed multiple times when the vehicle is moving. However, due to the influence of segmentation noise, the same area may be divided into multiple classes. To solve this problem, the present invention adopts a statistical method to filter the noise. The map is divided into small grids with a resolution of 0.1×0.1×0.1m. Each grid contains position, semantic label, and count information for each label. The semantic labels include ground, lane line, stop line, ground sign, and crosswalk. At initialization, the count value of each label is zero. When a semantic point is inserted into the grid, the count value of the corresponding label increases. Finally, the semantic label with the highest score represents the class of the grid. In this way, the semantic mapping can be kept accurate and is not affected by segmentation noise.

[0024] 5. Map Upload and Merging

[0025] The present invention aims to establish a global map of the autonomous driving area using ordinary civilian vehicles (rather than specialized mapping vehicles). The area covered by a single vehicle's driving is often not the entire area. Therefore, the present invention chooses to upload the single-vehicle map to the cloud for merging to form a global map. The cloud-stored map has the same format as the single-vehicle map, which is also a grid of 0.1×0.1×0.1m. In addition, there will be overlapping areas among the maps of multiple vehicles. For this problem, the present invention directly covers the overlapping areas to determine the accuracy of the map. That is: given the low-earth orbit satellite signal value at the starting point of the i-th single-vehicle map the starting coordinates (x j , y j ) of the j-th single vehicle, if there exists:

[0026] (x j , y j ) < (x s j , y s j ) (6)

[0027] then perform map merging to obtain a global high-precision semantic map.

[0028] The above are the steps for the present invention to construct a high-precision semantic map. The map update steps of the present invention are as follows:

[0029] 1. Real-time Perception and Localization of a Single Vehicle

[0030] To avoid incorrect map updates caused by real-time perception uncertainty, this invention abandons traditional perception-based map updating methods and instead utilizes perception-based positioning errors as a basis for determining map updates. Based on the semantic map constructed in the above steps, real-time perception and positioning steps are performed simultaneously with real-time mapping to ensure real-time map accuracy.

[0031] First, a semantic segmentation image is obtained from the forward real-time perception image. The semantic segmentation image is projected into the vehicle coordinate system according to the projection relationship (Formula 1) to obtain real-time feature points.

[0032] Then the real-time positioning information of the current vehicle is estimated by matching the current feature points with the map. The positioning adopts the ICP method, which can be written as the following equation:

[0033]

[0034] Among them, q and p are the quaternion and the position information of the current frame respectively; q * ,p * Represents the quaternion obtained by the vehicle based on map matching and the position information of the current frame. S is the set of current feature points, is the current k-th feature captured in the vehicle coordinate system, is the closest point of the image feature on the map in the global coordinate system.

[0035] Finally, the extended Kalman filter (EKF) framework is used to fuse the IMU odometer with the visual real-time positioning results. The filter not only increases the robustness of positioning, but also, as proven by experiments, can improve the positioning error caused by perception uncertainty. The fusion process is as follows:

[0036] (x i * ,y i * )=(q * ,p * )[RT] (8)

[0037] (x,y,θ)=EKF(x * ,y * ,x i * ,y i * ,θ i * ) (9)

[0038] Among them, (x i * ,y i * ) represents the position coordinate of the transformed world coordinate system, θ *It represents the vehicle orientation information output by the IMU, and [R T] is the transformation matrix from the local coordinate system (i.e., the vehicle coordinate system) to the world coordinate system. (x * , y * ) is the positioning information output by visual positioning, and EKF(·) is the Kalman filter function. Thus, the real-time positioning information (x, y, θ) in the global coordinates is obtained.

[0039] 2. Map Update Judgment

[0040] The following describes how to establish a map update model based on the positioning result as the judgment basis. In the present invention, the prior positioning information of the low-orbit satellite is used as the true value of the vehicle's real-time positioning, and (x, y, θ) obtained by the above formula (9) is used as the perceived positioning value. The present invention judges whether to update the map by comparing the relationship between the two. The present invention uses game theory to solve this problem, and the specific steps are as follows:

[0041] For this game task, the present invention defines whether to update the high-precision map as the system, which consists of two action options, namely {A1: Update the map, A2: Do not update the map}. Regarding the magnitude of the positioning error, the present invention defines it as the environment, which affects the optimality of the decision-making, and it has two states, namely {E1: The positioning difference is less than the threshold, E2: The positioning difference is greater than the threshold}.

[0042] Subsequently, the utility function is defined, which is divided into four scenarios: 1. Do not update the map when the positioning error is less than the threshold; 2. Update the map when the positioning error is less than the threshold; 3. Do not update the map when the positioning error is greater than the threshold; 4. Update the map when the positioning error is greater than the threshold; Calculate the utility of the system and the environment according to whether the system updates the map and the error magnitude. The utility functions U1 and U2 are specifically defined as follows:

[0043] U1(A1, E1) = -C1, U1(A2, E1) = 0 (10)

[0044] U1(A1, E2) = R1, U1(A2, E2) = -C2 (11)

[0045] U2(A1, E1) = +R2, U2(A2, E1) = 0 (12)

[0046] U2(A1, E2) = +R3, U2(A2, E2) = -C3 (13)

[0047] Among them, C1 is the cost of updating the map, R1 is the benefit brought by updating the map, assuming this is a positive value, C2 is the loss caused by not updating the map, R2 is the benefit brought to the environment by updating the map, C3 is the loss brought by updating the map, and R3 is the benefit brought to the environment by not updating the map. For this, the present invention constructs a payoff matrix:

[0048]

[0049] Since the positioning error may be random, the present invention may assume that the environment (the magnitude of the positioning error) is determined by a certain probability distribution. Therefore, the strategies of the system and the environment can be mixed strategies, indicating the probabilities of them choosing a certain behavior in each decision. The present invention makes the following assumptions: l is the probability that the error is E2 (i.e., the error is large), then the probability that the error is E1 is 1 - l. The probability that the system chooses to update the map (A1) is t, and the probability that it chooses not to update the map (A2) is 1 - t. Then the expected utility of the system is calculated as follows:

[0050] EU1(l) = l·t·R1 + l·(1 - l)·(-C2) + (1 - l)·t·(-C1) (15)

[0051] The expected utility of the environment is:

[0052] EU2(t) = l·t·R3 + l·(1 - t)·(-C3) + (1 - l)·t·R2 (16)

[0053] The optimal strategy is to maximize the expected utilities of the system and the environment. By taking the derivatives of (15) and (16) respectively and setting them equal to zero, the optimal strategies are obtained respectively and According to and the present invention can determine the optimal strategy of the system. For the optimal behavior of the environment, according to The system selects to update the map with a probability of and the remaining probability selects not to update the map. For the optimal behavior of the system, according to The system selects to update the map with a probability of and the remaining probability selects not to update the map. The selection of the threshold of the present invention is based on the actual average positioning error of the vehicle, and the threshold is set to be 50% beyond the actual positioning error.

[0054] Advantages of the present invention:

[0055] 1. To solve the problem that the high-precision map is vulnerable to the urban canyon effect, resulting in map trajectory drift, the present invention proposes to use low-orbit satellite signals as an accurate positioning information source to improve the map trajectory drift problem. The mapping algorithm of the present invention can be used for full coverage in urban areas.

[0056] 2. To obtain a precise map trajectory, the present invention optimizes the map by combining the IMU odometer with low-orbit satellite signals, improving the accuracy of the high-precision map.

[0057] 3. To improve the problem of incorrect map updates caused by perception uncertainty, the present invention proposes to use positioning errors to determine whether to update the map. Map errors are established using the positioning information provided by low-earth orbit satellites as prior information. Further, the present invention introduces game theory to determine whether to update the map, reducing the probability of incorrect map updates once again. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is the mapping process of the present invention;

[0059] Figure 2 is the map optimization of the present invention;

[0060] Figure 3 is the map update process of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0061] With the continuous expansion of the scale of commercial vehicle autonomous driving, the intelligence of commercial vehicles has become one of the hotspots in the development of automobiles today. High-precision maps are an important part of autonomous driving strategies, and perception, positioning, and path planning all rely on the prior information provided by high-definition maps. Previous work often used high-cost lidar and RTK to map high-precision maps. However, lidar sensors are expensive and difficult to popularize, and RTK is easily blocked in urban road scenarios, resulting in trajectory drift. At the same time, in map updates, previous work only used perception information as the basis for map updates. However, camera sensors have perception uncertainties, which will lead to incorrect map updates and map inaccuracies. Therefore, the present invention mainly proposes a method for constructing and updating a lightweight semantic map for autonomous driving based on low-earth orbit satellite prior information and perception uncertainty, using only camera sensors to produce and update high-precision semantic maps. For high-precision semantic map construction, it includes: S1. Image semantic segmentation S2. Inverse perspective transformation S3. Pose graph optimization S4. Single-vehicle mapping. S5. Map merging and production. For high-precision semantic map updates, it includes: S1. Single-vehicle real-time perception and positioning S2. Map update judgment. This method mainly solves the problem of constructing and updating high-precision semantic maps for urban road autonomous driving. In map construction, to address the problem that RTK signals are easily blocked by buildings and trees, it is proposed to use low-earth orbit satellites as trajectory signals, and at the same time optimize the map using RTK signals and IMU odometers for more accurate trajectory recovery. In map updates, to address the problem of perception uncertainty of camera sensors, it is proposed to use real-time positioning errors as the basis for updating the map, use low-earth orbit satellite signals as positioning ground truths, and use game theory to determine whether to update the map, further reducing the problem of incorrect updates caused by camera perception uncertainty.

[0062] The present invention will be further described below with reference to the accompanying drawings.

[0063] As Figure 1 shown, the specific implementation of the high-precision map establishment of the present invention is described as follows:

[0064] As described in step 1: Image semantic segmentation. The vehicle is equipped with a monocular camera, and the real-time image is obtained and fed into the DeepLabV3+ semantic segmentation model to obtain semantic images of four categories: ground, lane lines, stop lines, and road markings. At the same time, the elements of the semantic map are also obtained.

[0065] As described in step 2: Inverse perspective transformation. After obtaining the semantic image, the present invention performs inverse perspective transformation on the semantic image pixel by pixel, converting the semantic image pixel coordinates into the global coordinates of the high-precision map. After obtaining the coordinates of each pixel in the semantic image, each pixel is projected into the map coordinate system according to the vehicle calibration result to obtain the semantic map. Specifically:

[0066] The internal parameters of the camera and the external parameters from the camera to the vehicle center coordinate origin are calibrated offline. Since there is a mapping error in the inverse perspective mapping and the error increases with the increase of the scene distance, the present invention artificially sets the projection mapping distance. According to a large number of experiments, it is finally determined that the mapping range is a rectangular area of 12m × 6m. Experiments have proved that if this area is exceeded, the mapping error will increase instantaneously. The present invention assumes that the urban road is a plane, and the image pixel coordinates in the image coordinate system after semantic segmentation are (u, v). It is assumed that the image pixel coordinates are projected onto the ground to obtain the coordinates (x u , y v ), then:

[0067]

[0068] Among them, π c (.) is the camera undistortion and projection model, and π c (.) -1 represents inverse projection, that is, lifting the image pixels to the world coordinate system space. [R c t c is the external parameter matrix of each image pixel relative to the vehicle center. R c is the rotation matrix of the image pixel relative to the vehicle center, and t c is the translation matrix of the image pixel relative to the vehicle center. λ is a scalar. [] col:1，2，4 means taking the 1st, 2nd, and 4th columns of the matrix.

[0069] As described in step 3: Pose graph optimization. After obtaining the individual frame maps, the present invention restores the trajectories between each frame of the map. As Figure 2 shown, the low-orbit satellite signal and the output result of the vehicle IMU odometer are fed into the graph optimization model together to output the pose information, and then each frame of the map is connected according to the pose information to form a continuous and complete map. Specifically:

[0070] Let the state of the vehicle at a certain moment be s, which includes the position p and the orientation q represented by a quaternion. Use the operation R(q) to convert the quaternion into a rotation matrix. For graph optimization, there are two types of edges. One is the low-earth orbit satellite signal, and the other is the odometer signal. These two types of edges exist at any time. The pose graph optimization can be expressed as:

[0071]

[0072] where s represents the vehicle pose state (position and orientation). r o is the residual factor of the odometer signal. is the odometer measurement, which includes the position increment and the orientation increment r g is the residual of the low-earth orbit satellite factor. is the low-earth orbit satellite measurement, that is, the position in the world coordinate system The residual factors r o and r g are defined as follows:

[0073]

[0074]

[0075] where p i represents i the vehicle position at time i q represents i the vehicle orientation at time i-1 ), and R(q i-1 ) is the rotation matrix corresponding to the vehicle orientation at time xyz [.] represents taking the first three elements of the quaternion, approximately equal to an error perturbation on the manifold. A manifold is a mathematical concept referring to a low-dimensional structure or surface in a high-dimensional space.

[0076] As described in step 4: single vehicle mapping. The map after restoring the trajectory is not directly uploaded to the cloud. The present invention grids it to reduce the map occupancy space. The present invention assigns accurate pose, category, and category score to each grid, thereby accurately dividing the original map into a grid map. Among them, due to the influence of segmentation noise, the same area may be divided into multiple categories. To solve this problem, the present invention uses statistical methods to filter the noise. Specifically as follows:

[0077] The map is divided into small grids with a resolution of 0.1×0.1×0.1 m. Each grid contains position, semantic labels, and the count information for each label. The semantic labels include ground, lane lines, stop lines, ground signs, and crosswalks. At initialization, the count value for each label is zero. When a semantic point is inserted into the grid, the count value of the corresponding label is incremented. Finally, the semantic label with the highest score represents the category of the grid. In this way, the semantic mapping can remain accurate and is not affected by segmentation noise.

[0078] As described in step 5: Map upload and merging. The grid semantic maps completed by each vehicle are uploaded to the cloud and stored by the cloud. After each single-bike map is uploaded to the cloud, the cloud merges the maps according to the same initial global pose of each map to form a complete high-precision semantic map of the city. Among them, for the problem that the maps of multiple single bikes will contain overlapping areas, the present invention directly covers the overlapping areas to determine the accuracy of the map, that is: given the coordinates of the low-orbit satellite signal at the starting point of the i-th single-bike map The starting coordinates (x j , y j ) of the j-th single bike, if there exists:

[0079] (x j , y j ) < (x s j , y s j ) (6)

[0080] Then map merging is performed to obtain a global high-precision semantic map.

[0081] As Figure 3 shown, the specific implementation steps of the high-precision map update of the present invention are as follows:

[0082] As described in step 1: Real-time perception and positioning of a single bike. After the vehicle travels into the high-precision map environment, it perceives the image in real time, performs semantic segmentation, and executes the inverse perspective operation to obtain real-time perception features. At the same time, the above-mentioned single-bike mapping steps are executed to prevent missed map updates. Subsequently, real-time positioning is performed, and the perception features are used to perform ICP positioning with the cloud map to output the positioning result. The positioning result and the IMU odometer result are sent to the Kalman filter together to output the real-time positioning result. Specifically as follows:

[0083] Obtain a semantic segmentation image from the front-view real-time perception image, and project the semantic segmentation image onto the vehicle coordinate system according to the projection relationship to obtain real-time feature points;

[0084] Estimate the real-time positioning information of the current vehicle by matching the current feature points with the map, and adopt the ICP positioning method, as follows:

[0085]

[0086] where q and p are the quaternion and the position information of the current frame respectively; is the set of current feature points of S, is to capture the current feature in the vehicle coordinate system, is the nearest point of this feature on the map in the global coordinate system;

[0087] Finally, the odometer and the visual positioning results are fused using the Extended Kalman Filter (EKF) framework. The fusion process is described as follows:

[0088] (x i * , y i * ) = (q * , p * )[R T] (8)

[0089] (x, y, θ) = EKF(x * , y * , x i * , y i * , θ i * ) (9)

[0090] where (x i * , y i * ) represents the position coordinates in the converted world coordinate system, θ * represents the vehicle orientation information output by the IMU, and [R T] is the transformation matrix from the local coordinate system to the world coordinate system; (x * , y * ) is the positioning information output by visual positioning, and EKF(·) is the Kalman filter function; (x, y, θ) is the real-time positioning information in the global coordinates.

[0091] As described in step 2: Map update judgment. After obtaining the real-time positioning information, the difference between the real-time positioning information and the real-time signal of the low-earth orbit satellite positioning is calculated, and the difference is input into the game theory model. Subsequently, after game calculation, the optimal map update strategy is obtained. Specifically as follows:

[0092] Define whether to update the high-precision map as the system, which consists of two action options, namely {A1: Update the map, A2: Do not update the map}; for the magnitude of the positioning error, define it as the environment, which affects the optimality of the decision-making, and it has two states, namely {E1: The positioning difference is less than the threshold, E2: The positioning difference is greater than the threshold};

[0093] Subsequently, the utility function is defined to calculate the utility of the system and the environment based on whether the system updates the map and the magnitude of the error. The utility function is defined as follows:

[0094] U1(A1,E1) = -C1, U1(A2,E1) = 0 (10)

[0095] U1(A1,E2) = R1, U1(A2,E2) = -C2 (11)

[0096] U2(A1,E1) = +R2, U2(A2,E1) = 0 (12)

[0097] U2(A1,E2) = +R3, U2(A2,E2) = -C3 (13)

[0098] Where C1 is the cost of updating the map; R1 is the benefit brought by updating the map, which is a positive value; C2 is the loss caused by not updating the map, R2 is the benefit brought by updating the map to the environment, R3 is the benefit brought by updating the map to the environment, and C3 is the loss caused by not updating the map;

[0099] Construct the payoff matrix:

[0100]

[0101] Since the positioning error may be random, assuming that the magnitude of the positioning error is determined by a certain probability distribution. Therefore, the strategies of the system and the environment can be mixed strategies, indicating the probability of choosing a certain behavior in each decision. Make the following assumptions: l is the probability of the error being E2 (i.e., the error is large), then the probability of the error being E1 is 1 - l, the probability of the system choosing to update the map is t, and the probability of choosing not to update the map is 1 - t,

[0102] Then the expected utility of the system is calculated as:

[0103] EU1(l) = l·t·R1 + l·(1 - l)·(-C2) + (1 - l)·t·(-C1) (15)

[0104] The expected utility of the environment is:

[0105] EU2(t) = l·t·R3 + l·(1 - t)·(-C3) + (1 - l)·t·R2 (16)

[0106] The optimal strategy is to maximize the expected utilities of the system and the environment. Take the derivatives of (15) and (16) respectively and set them equal to zero to obtain the optimal strategies and According to and Determine the optimal strategy of the system. For the optimal behavior of the environment, according to That is, the system updates the map with probability and does not update the map with the remaining probability ; For the optimal behavior of the system, according to That is, the system updates the map with probability and does not update the map with the remaining probability ;

[0107] The series of detailed descriptions listed above are only specific descriptions of the feasible implementation manners of the present invention, and they are not intended to limit the protection scope of the present invention. Any equivalent manners or modifications that do not depart from the technical creation of the present invention should be included in the protection scope of the present invention.

Claims

1. A lightweight semantic map construction method for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty, characterized in that It includes the following: S1 Image semantic segmentation; S2 Inverse perspective transformation, which projects the segmented semantic pixels from the image plane onto the ground plane in the vehicle coordinate system; S3 Perform graph optimization on the vehicle's pose; S4 Single-vehicle mapping; S5 Upload the single-vehicle map to the cloud for merging to obtain a global map.

2. A lightweight semantic map construction method for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty according to claim 1, characterized in that The implementation of S1 includes: The real-time image obtained by the vehicle's monocular camera is input into the DeepLabV3+ semantic segmentation model to obtain semantic images of four categories: ground, lane lines, stop lines, and road markings, which are also used as elements of the semantic map.

3. A lightweight semantic map construction method for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty according to claim 1, characterized in that, The implementation of S2 includes: S2.1 Calibrate the internal parameters of the vehicle's monocular camera and the external parameters from the camera to the vehicle's center coordinate origin offline; S2.2 Since there is a mapping error in the inverse perspective mapping and the error increases with the increase of the scene distance, a projection mapping distance is artificially set, and the mapping range is determined to be a 12m×6m rectangular area; S2.3 Assume that the urban road is a plane. The pixel coordinates of the image in the image coordinate system after semantic segmentation are (u, v). The pixel coordinates of the image are projected onto the ground to obtain the coordinates (x u ,y v ),but: where, π c (.) is camera undistortion and projection transformation, π c (.) -1 represents inverse projection, that is, lifting image pixels to the world coordinate system space. [R c t c is the extrinsic parameter matrix of each image pixel relative to the vehicle center, R c is the rotation matrix of the image pixel relative to the vehicle center, t c is the translation matrix of the image pixel relative to the vehicle center, λ is a scalar, col:1，2，4 denotes taking the 1st, 2nd, and 4th columns of the matrix.

4. A lightweight semantic map construction method for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty according to claim 1, characterized in that The implementation of S3 includes: S3.1 Introduce low-earth orbit satellite signals as optimization factors, and low-earth orbit satellite signals provide accurate vehicle pose information globally; S3.2 Assume the state of the vehicle at a certain moment is s, which includes the position p and the direction q represented by a quaternion. Use operations to convert the quaternion into a rotation matrix. For graph optimization, there are two types of edge signals, one is the low-earth orbit satellite signal, and the other is the odometer signal. Both types of edges exist at any time. The pose graph optimization can be expressed as: Among them, s represents the vehicle posture state, r o is the residual factor of the odometry signal, is the odometry measurement, which contains the position increment between two adjacent states With direction increment r g is the residual factor of the low-orbit satellite signal, is the measurement value of the low-orbit satellite, that is, the position in the world coordinate system Residual factor r o and r g The definition is as follows: Among them, [.] xyz represents taking the first three elements of the quaternion, which is approximately equal to the error perturbation on the manifold, p i represents the vehicle position at the i-th moment, q i represents the vehicle direction at the i-th moment, R(q i-1 ) is the rotation matrix corresponding to the vehicle direction at the (i - 1)-th moment.

5. A method for constructing a lightweight semantic map for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty according to claim 1, characterized in that The implementation of S4 includes: S4.1 Each frame of semantic image i is converted from the vehicle coordinate system to the global coordinate system according to the optimized pose, that is: x w 、y w 、z w represent the coordinates of the global coordinate system; S4.2 Through image segmentation, each point is assigned a class label. The position of each point in the world coordinate system corresponds to a small area. To address the problem that the same area may be divided into multiple categories, a statistical method is used to filter segmentation noise, specifically as follows: Divide the map into small grids with a resolution of 0.1×0.1×0.1m. Each grid contains position, semantic label, and the count information of each label. The semantic labels include ground, lane lines, stop lines, ground signs, and crosswalks. When initializing, the count value of each label is zero. When a semantic point is inserted into the grid, the count value of the corresponding label will increase. Finally, the semantic label with the highest score represents the category of the grid.

6. A lightweight semantic map construction method for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty according to claim 1, characterized in that The implementation of S5 includes: Upload the bicycle maps to the cloud for merging to form a global map, which has the same format as the bicycle maps. The cloud-stored map is also a grid of 0.1×0.1×0.1m; for the problem that the multiple bicycle maps may contain overlapping areas, directly cover the overlapping areas to determine the map accuracy, that is: given the low-orbit satellite signal value at the starting point of the i-th bicycle map The starting coordinates (x j , y j ) of the j-th bicycle, if there exists: (x j ,y j ) < (x s j ,y s j ) (6) Then perform map merging to obtain a global high-precision semantic map.

7. A method for updating a lightweight semantic map for autonomous driving based on prior information of low-earth orbit satellites and perception uncertainty, characterized in that, The lightweight semantic map is constructed by the method described in any one of claims 1-6; the update method includes: S6 Single-vehicle real-time perception and positioning; S7 Use the positioning result as the judgment basis for map update.

8. The method for updating an autonomous driving lightweight semantic map according to claim 7, wherein The implementation of S6 includes: S6.1 Obtain a semantic segmentation image from the front-view real-time perception image, and project the semantic segmentation image onto the vehicle coordinate system according to the projection relationship to obtain real-time feature points; S6.2 Estimate the real-time positioning information of the current vehicle by matching the current feature points with the map. The positioning method is described as follows: where q and p are the quaternion and the position information of the current frame respectively; is the set of current feature points of S, is to capture the current feature in the vehicle coordinate system, is the nearest point of this feature on the map in the global coordinate system; S6.3 Finally, use the extended Kalman filter EKF framework to fuse the odometer and visual positioning results. The fusion process is described as follows: (x i * , y i * ) = (q * , p * ) [R T] (8) (x,y,θ)=EKF(x * ,and * ,x i * ,and i * ,θ i * ) (9) Among them, (x i * , y i * ) represents the position coordinates in the converted world coordinate system, θ * represents the vehicle direction information output by the IMU, and [R T] is the transformation matrix from the local coordinate system (i.e., the vehicle coordinate system) to the world coordinate system. (x * , y * ) is the positioning information output by visual positioning, and EKF(·) is the Kalman filter function. Thus, the real-time positioning information (x, y, θ) in the global coordinates is obtained.

9. The updating method of the lightweight semantic map for autonomous driving according to claim 8, wherein The implementation of S7 includes: Using the prior positioning information of low-orbit satellites as the true value of vehicle real-time positioning, and taking (x, y, θ) obtained by the above formula (9) as the perceived positioning value, the map is judged whether to be updated by comparing the relationship between the two, and the game theory is used for solution.

10. The method for updating the lightweight semantic map for autonomous driving according to claim 9, wherein The specific implementation of the solution using the game theory is as follows: Define whether to update the high-precision map as the system, which consists of two action options, namely {A1: Update the map, A2: Do not update the map}; for the magnitude of the positioning error, it is defined as the environment, which affects the optimality of the decision-making, and it has two states, namely {E1: The positioning difference is less than the threshold, E2: The positioning difference is greater than the threshold}; Subsequently, define the utility function, and calculate the utility of the system and the environment according to whether the system updates the map and the error magnitude. The utility function is defined as follows: U1(A1, E1) = -C1, U1(A2, E1) = 0 (10) U1(A1, E2) = R1, U1(A2, E2) = -C2 (11) U2(A1, E1) = +R2, U2(A2, E1) = 0 (12) U2(A1, E2) = +R3, U2(A2, E2) = -C3 (13) Among them, C1 is the cost of updating the map; R1 is the benefit brought by updating the map, which is a positive value; C2 is the loss caused by not updating the map, R2 is the benefit brought by updating the map to the environment, C3 is the loss brought by updating the map, and R3 is the benefit brought by not updating the map to the environment; Construct the payoff matrix: Since the positioning error may be random, assume that the magnitude of the positioning error is determined by a certain probability distribution. Therefore, the strategies of the system and the environment can be mixed strategies, indicating the probability of choosing a certain behavior in each decision-making. Make the following assumptions: l is the probability that the error is E2 (that is, the error is large), then the probability that the error is E1 is 1 - l, the probability that the system chooses to update the map (A1) is t, and the probability of choosing not to update the map (A2) is 1 - t. Then calculate the expected utility of the system as: EU1(l) = l·t·R1 + l·(1 - l)·(-C2) + (1 - l)·t·(-C1) (15) The expected utility of the environment is: EU2(t) = l·t·R3 + l·(1 - t)·(-C3) + (1 - l)·t·R2 (16) The optimal strategy is to maximize the expected utility of the system and the environment. By taking the derivatives of (15) and (16) respectively and setting them equal to zero, the optimal strategies are obtained and According to and to determine the optimal strategy of the system. For the optimal behavior of the environment, according to that is, the system updates the map with probability and the remaining probability chooses not to update the map; for the optimal behavior of the system, according to that is, the system updates the map with probability and the remaining probability chooses not to update the map.

Images