Map determination method, device, vehicle and medium

Abstract

The present application relates to a map determination method, device, vehicle, and medium. The method includes at least: determining target image features and target point cloud features of multiple target objects in the vehicle's driving environment based on image data and point cloud data of the vehicle during driving; constructing a three-dimensional visual map of the vehicle based on the target image features of the multiple target objects and the first positions of the multiple target objects; constructing a three-dimensional laser map of the vehicle based on the target point cloud features of the multiple target objects and the second positions of the multiple target objects; determining a target map for assisting vehicle driving based on at least the three-dimensional visual map and the three-dimensional laser map; storing the target map, determining stable areas in the target map, and storing a first validity period for the stable areas and a second validity period for the unstable areas in the target map. The target map determined by this solution is more accurate.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a map determination method, device, vehicle, and medium. Background Art

[0002] With the rapid development of autonomous driving and intelligent transportation technologies, how to obtain an accurate map has become one of the key factors for whether intelligent vehicles can operate normally in complex environments.

[0003] In related technologies, visual maps are generally constructed directly based on collected image data. However, in complex scenarios, such as rainy or foggy conditions, the accuracy of the visual map is low, which may affect vehicle driving and reduce vehicle safety. Summary of the Invention

[0004] One of the purposes of this application is to provide a map determination method, device, vehicle and medium. In this solution, a three-dimensional visual map and a three-dimensional laser map are constructed, and a target map is generated based on the three-dimensional visual map and the three-dimensional laser map, thereby improving the accuracy of the target map and the safety of driving, and improving the effectiveness of the target map display.

[0005] In order to achieve the above objectives, the technical solutions adopted in this application are as follows:

[0006] In a first aspect, the present application provides a map determination method, the method comprising: based on image data and point cloud data of a vehicle during driving, respectively determining target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle driving environment; constructing a three-dimensional visual map of the vehicle based on the target image features of the multiple target objects and the first positions of the multiple target objects; the first position is obtained based on the image data; constructing a three-dimensional laser map of the vehicle based on the target point cloud features of the multiple target objects and the second positions of the multiple target objects; the second position is obtained based on the point cloud data; determining a target map for assisting vehicle driving based on at least the three-dimensional visual map and the three-dimensional laser map; storing the target map, determining stable areas and unstable areas in the target map, and storing a first validity period of the stable area and a second validity period of the unstable area in the target map; the first validity period is greater than the second validity period.

[0007] Based on the above technical means, a 3D visual map is first constructed from image data, and a 3D laser map is constructed from point cloud data. The target map is then determined based on the 3D visual map and the 3D laser map. This combines the advantages of 3D visual maps and 3D laser maps, improving the accuracy and application scenarios of the target map and enhancing vehicle safety. Furthermore, the validity period of different areas can be clearly displayed, enhancing the effectiveness of the target map display.

[0008] In one possible implementation, a target map for assisting vehicle driving is determined based at least on a three-dimensional visual map and a three-dimensional laser map, including: determining the target map based on the three-dimensional visual map and the three-dimensional laser map; or determining a target local map based on the three-dimensional visual map and the three-dimensional laser map; determining the target map based on the target local map and the global map of the vehicle.

[0009] Based on the above technical means, when determining the target map, if the target map is determined directly based on the three-dimensional visual map and the three-dimensional laser map, the advantages of both can be combined to improve the accuracy of the local map; when determining the target map, if it is determined based on the vehicle's global map, three-dimensional visual map and three-dimensional laser map, not only the accuracy of the local map is improved, but also the global planning is improved, the comprehensiveness of the target map is improved, and the accuracy of driving is improved.

[0010] In one possible implementation, a target local map is determined based on a three-dimensional visual map and a three-dimensional laser map, including: when a first condition is met, determining the three-dimensional visual map as the target local map; when the first condition is not met, determining the three-dimensional laser map as the target local map; wherein the first condition is used to characterize that the light intensity in the vehicle driving environment meets a light intensity threshold.

[0011] Based on the above technical means, the target local map is determined based on the light intensity in the vehicle's driving environment. When the light intensity meets the intensity threshold, the image information is more comprehensive and the resulting visual map is more accurate. Therefore, the target local map determined based on the three-dimensional visual map is also accurate. When the light intensity does not meet the intensity threshold, the image information is low, the resulting visual map has more loss and is less accurate. Therefore, the target local map is determined based on the three-dimensional laser map. The target map obtained in this way can meet the needs of various scenarios, improving accuracy and application scenarios.

[0012] In one possible embodiment, a target local map is determined based on a three-dimensional visual map and a three-dimensional laser map, including: determining a first position of each target object in the three-dimensional visual map; determining a second position of each target object in the three-dimensional laser map; fusing the first position and the second position through a filter to obtain a target position of each target object; and determining a target local map based on a target feature of each target object and the target position of each target object; wherein the target feature is obtained based on image features and / or point cloud features.

[0013] Based on the above technical means, when determining the local map of the target, the three-dimensional visual map and the three-dimensional laser map can also be fused. Since the map generally includes objects and their positions, fusing the position of each object improves the accuracy of the position, and fusing the features improves the accuracy of the features.

[0014] In a possible implementation, the method further includes: identifying the image data to obtain at least one static object and at least one dynamic object; filtering the identified at least one dynamic object and determining at least one static object as the target object.

[0015] Based on the above technical means, dynamic objects are filtered out. Since dynamic objects have dynamic errors, they will affect the accuracy of the map. The three-dimensional map determined based on static objects is more accurate.

[0016] In a possible embodiment, when the target objects include static objects and dynamic objects, a three-dimensional visual map of the vehicle is constructed based on the target image features of multiple target objects and the first positions of multiple target objects, including: determining the first influence coefficient of the static object and the second influence coefficient of the dynamic object respectively; the first influence coefficient is greater than the second influence coefficient; and constructing the three-dimensional visual map based on the target image features, first position and first influence coefficient of the static object, and the target image features, first position and second influence coefficient of the dynamic object.

[0017] Based on the above technical means, when the target objects include static objects and dynamic objects, by configuring the first influence coefficient and the second influence coefficient, the influence of static objects on the three-dimensional map is increased, and the influence of dynamic objects on the three-dimensional map is reduced, and the obtained three-dimensional map is more accurate.

[0018] In one possible implementation, target image features of multiple target objects in a vehicle environment are determined based on image data of a moving vehicle, including: recognizing the image data of the moving vehicle through a first network model to obtain initial feature maps of the multiple target objects; performing target-scale pooling processing on the initial feature maps of the multiple target objects through a second network model to obtain feature maps of the target scale; and splicing the feature maps of the target scale with the initial feature maps to obtain target image features of the multiple target objects.

[0019] Based on the above technical means, when determining the target image features, the target scale can be configured based on actual needs, so that the target image features that meet the actual needs can be obtained.

[0020] In a possible embodiment, when the target scale includes multiple scales, the target image features include target image features at multiple scales; correspondingly, the three-dimensional visual map includes three-dimensional visual maps of multiple resolutions; the target map includes target maps of multiple resolutions; the method also includes: when the second condition is met, assisting vehicle driving based on the target map of the first resolution; when the third condition is met, assisting vehicle driving based on the target map of the second resolution; wherein the first resolution is lower than the second resolution, the second condition is used to characterize that the vehicle's driving environment is an open environment; and the third condition is used to characterize that the vehicle's driving environment is a narrow environment or a complex environment.

[0021] Based on the above technical means, target maps with multiple resolutions can be obtained. Target maps with different resolutions can be switched in different driving scenarios to meet the needs of various scenarios.

[0022] In one possible embodiment, based on the image data and point cloud data of the vehicle while driving, target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle's driving environment are respectively determined, including: obtaining initial image data and initial point cloud data of the vehicle while driving; filtering the initial image data and initial point cloud data to obtain first image data and first point cloud data; detecting and correcting abnormal data in the first image data and the first point cloud data to obtain corrected second image data and second point cloud data; temporally aligning the second image data and the second point cloud data using a time interpolation technique to obtain aligned third image data and third point cloud data; and determining target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle's driving environment based on the third image data and the third point cloud data of the vehicle while driving.

[0023] Based on the above technical means, the initial image data and initial point cloud data can be filtered, anomaly corrected, time-aligned, etc., so that the processed data can better meet the needs, improve the accuracy of the data, and thus improve the accuracy of the target map and driving safety.

[0024] In one possible embodiment, the method also includes: constructing a first node based on the vehicle posture and environmental features of the vehicle at each moment; the environmental features are obtained based on image features and / or point cloud features; determining the first constraint relationship between adjacent first nodes as the matching degree between the environmental features of adjacent first nodes; constructing a time graph based on the first node and the first constraint relationship between the first nodes; adjusting the target map by adjusting the first node and the constraint relationship between the first nodes in the time graph.

[0025] Based on the above technical means, the target map can be adjusted. Adjustment based on the time map can reduce the time accumulation error, thereby improving the accuracy of the target map and driving safety.

[0026] In one possible embodiment, the method also includes: constructing a second node based on different positions of the vehicle; determining the edge between each two second nodes as the posture change value between each two second nodes; constructing a pose graph based on the second node and the edge between the second node; adjusting the second node and the edge between the second nodes in the pose graph, and adjusting the target map.

[0027] Based on the above technical means, the target map can be adjusted. Adjustments based on the pose graph can reduce the cumulative error caused by position changes, thereby improving the accuracy of the target map and driving safety.

[0028] In a possible implementation, the method further includes: when it is detected that the second validity period of the unstable area has reached, reconstructing and updating the unstable area in the target map.

[0029] Based on the above technical means, the map information of unstable areas can be updated in a timely manner, achieving high precision of the target map while also having high real-time performance.

[0030] In a possible implementation, the method further includes: determining predicted behaviors of multiple target objects; if the predicted behaviors of the target objects affect the driving of the vehicle, outputting a first prompt message on the target map; the first prompt message is used to prompt that the predicted behaviors of the target objects affect the driving of the vehicle.

[0031] Based on the above technical means, behaviors that affect vehicle driving can be output in the target map, which improves the richness of the target map display and the safety of vehicle driving.

[0032] In a second aspect, the present application provides a map determination device, the device comprising:

[0033] a first determining unit, configured to respectively determine target image features of a plurality of target objects and target point cloud features of a plurality of target objects in a driving environment of the vehicle based on image data and point cloud data of the vehicle during driving;

[0034] A first construction unit is configured to construct a three-dimensional visual map of the vehicle based on target image features of the plurality of target objects and first positions of the plurality of target objects, wherein the first positions are obtained based on the image data;

[0035] A second construction unit is configured to construct a three-dimensional laser map of the vehicle based on target point cloud features of the plurality of target objects and second positions of the plurality of target objects; the second positions are obtained based on the point cloud data;

[0036] a second determining unit, configured to determine a target map for assisting vehicle driving based at least on the three-dimensional visual map and the three-dimensional laser map;

[0037] The storage unit is used to store the target map, determine the stable area and the unstable area in the target map, and store the first validity period of the stable area and the second validity period of the unstable area in the target map; the first validity period is greater than the second validity period.

[0038] In a third aspect, the present application further provides a vehicle, comprising a processor and a memory, wherein the memory stores a computer program or instructions, and when the computer program or instructions are executed by the processor, the method provided in the first aspect is implemented.

[0039] In a fourth aspect, the present application further provides a storage medium storing a computer program or instruction, which, when executed by a processor, implements the method provided in the first aspect.

[0040] In a fifth aspect, the present application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the method provided in the first aspect is implemented.

[0041] It should be noted that the technical effects of the second to fifth aspects can refer to the detailed description of the first aspect above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a first optional flow chart of a method for determining a map provided in an embodiment of the present application;

[0043] Figure 2 A second optional flow chart of the method for determining a map provided in an embodiment of the present application;

[0044] Figure 3 A third optional flow chart of the map determination method provided in the embodiment of the present application;

[0045] Figure 4 A fourth optional flow chart of the method for determining a map provided in an embodiment of the present application;

[0046] Figure 5 A fifth optional flow chart of the map determination method provided in the embodiment of the present application;

[0047] Figure 6 A sixth optional flow chart of the map determination method provided in the embodiment of the present application;

[0048] Figure 7A seventh optional flow chart of the map determination method provided in the embodiment of the present application;

[0049] Figure 8 This is a schematic diagram of an eighth optional flow chart of the map determination method provided in the embodiment of the present application;

[0050] Figure 9 A ninth optional flow chart of the method for determining a map provided in an embodiment of the present application;

[0051] Figure 10 A tenth optional flowchart of the map determination method provided in the embodiment of the present application;

[0052] Figure 11 This is a schematic diagram of an eleventh optional flow chart of the map determination method provided in the embodiment of the present application;

[0053] Figure 12 An optional structural diagram of the data processing related modules provided in the embodiment of the present application;

[0054] Figure 13 An optional flowchart of the positioning process provided in an embodiment of the present application;

[0055] Figure 14 An optional flowchart of the process of scene understanding provided in an embodiment of the present application;

[0056] Figure 15 A schematic diagram of an optional process for positioning and map construction provided in an embodiment of the present application;

[0057] Figure 16 An optional flowchart of the map optimization process provided in the embodiment of the present application;

[0058] Figure 17 An optional flowchart of the map storage process provided in the embodiment of the present application;

[0059] Figure 18 A schematic diagram of an optional structure of a map determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0061] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0062] In the following description, the terms "first, second, and third" are used merely as examples to distinguish between different objects and do not represent a specific order or precedence for the objects. It is understood that the specific order or precedence of "first, second, and third" can be interchanged where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0064] The present application provides a map determination method, apparatus, vehicle, medium, and product. The map determination method is performed by a map determination device, which can be deployed on a vehicle. The following describes various embodiments of the map determination method, apparatus, vehicle, medium, and product provided in the present application.

[0065] In a first aspect, an embodiment of the present application provides a method for determining a map.

[0066] refer to Figure 1 The process may include but is not limited to S101 to S105. The process may be implemented by a local device on the vehicle side, or by a cloud connected to the vehicle side, or by a combination of the vehicle side and the cloud side, and by interaction between the vehicle side and the cloud side.

[0067] S101 : Based on image data and point cloud data of a vehicle during driving, respectively determine target image features of a plurality of target objects and target point cloud features of a plurality of target objects in a vehicle driving environment.

[0068] The embodiments of this application do not limit the type of vehicle and can be configured according to actual needs. For example, vehicles may include but are not limited to: gasoline vehicles, electric vehicles, etc. The vehicles here can be vehicles with or without assisted driving functions.

[0069] Image data refers to data captured by cameras. Point cloud data refers to data collected by lidar or millimeter-wave radar. Image data and point cloud data can be raw data or processed (filtered, etc.)

[0070] The vehicle driving environment refers to the external environment during the vehicle driving process.

[0071] Target objects refer to objects in the vehicle's driving environment that meet the requirements. For example, target objects can be all objects or only static objects.

[0072] Image features are used to characterize object information in an image, and target image features are used to characterize target object information in an image.

[0073] Point cloud features are used to represent object information in the point cloud, and target image features are used to represent target object information in the point cloud.

[0074] S101 can be implemented as follows: after acquiring image data of a vehicle in motion, performing recognition processing on the image data using a target recognition algorithm to identify multiple target objects, and then performing feature extraction on the image data corresponding to the multiple target objects using a feature extraction algorithm to obtain target image features of the multiple target objects; after acquiring point cloud data of a vehicle in motion, performing recognition processing on the point cloud data using a target recognition algorithm to identify multiple target objects, and then performing feature extraction on the point cloud data of the multiple target objects using a feature extraction algorithm to obtain target point cloud features of the multiple target objects.

[0075] The recognition algorithm used in the image data processing process may be the same as or different from the recognition algorithm used in the point cloud data processing process. The feature extraction algorithm used in the image data processing process may be the same as or different from the feature extraction algorithm used in the point cloud data processing process.

[0076] S102: Construct a three-dimensional visual map of the vehicle based on target image features of the multiple target objects and the first positions of the multiple target objects.

[0077] The first position is obtained based on the image data. The embodiment of the present application does not limit the method of determining the first position based on the image data, and can be configured according to actual needs. For example, the first position can be determined based on image data from two different angles.

[0078] The target objects here include objects in the vehicle's driving environment, such as pedestrians, buildings, etc., and also the vehicle itself.

[0079] S102 can be implemented as follows: construct a three-dimensional visual map based on the first position of the vehicle, first obtain the first position of the vehicle and the target image features of the vehicle, and render the vehicle in the three-dimensional visual map at the first position of the vehicle based on the target image features of the vehicle; then obtain the first positions of other target objects and the target image features of other target objects, and render each other target object in the three-dimensional visual map at the first position of each other target object based on the image features of each other target object, thereby obtaining a three-dimensional visual map of the vehicle.

[0080] S103: Construct a three-dimensional laser map of the vehicle based on the target point cloud features of the multiple target objects and the second positions of the multiple target objects.

[0081] The second position is obtained based on the point cloud data. The embodiment of the present application does not limit the method of determining the second position based on the image data, and can be configured according to actual needs. For example, the second position can be determined based on point cloud data at two different times.

[0082] S103 can be implemented as follows: construct a three-dimensional laser map based on the second position of the vehicle, first obtain the second position of the vehicle and the target point cloud features of the vehicle, and render the vehicle in the three-dimensional laser map at the second position of the vehicle based on the target point cloud features of the vehicle; then obtain the second positions of other target objects and the target point cloud features of other target objects, and render each other target object in the three-dimensional laser map at the second position of each other target object based on the point cloud features of each other target object, thereby obtaining a three-dimensional laser map of the vehicle.

[0083] S104: Determine a target map for assisting vehicle driving based at least on the three-dimensional visual map and the three-dimensional laser map.

[0084] In a possible implementation, a target map for assisting vehicle driving is determined based on the three-dimensional visual map and the three-dimensional laser map. The target map obtained in this way is more accurate.

[0085] In another possible implementation, S104 may be implemented as follows: determining a target map for assisting vehicle driving based on the global map, the 3D visual map, and the 3D laser map. For example, a target local map may be first determined based on the 3D visual map and the 3D laser map, and then the target local map may be spliced ​​and fused with the global map to obtain the target map. This resulting target map is both accurate and comprehensive.

[0086] In another possible implementation, S104 may be implemented as follows: a target map for assisting vehicle driving may also be determined based on inertial data, a global map, a 3D visual map, and a 3D laser map. For example, a target local map may be first determined based on the 3D visual map and the 3D laser map. This target local map is then merged with the global map to produce a fused map. This fused map is then optimized based on the inertial data. The resulting target map can be used in weak network environments. For example, in a tunnel scenario, the vehicle's position can be determined based on inertial data to update the target map.

[0087] S105 , storing the target map, determining a stable area in the target map, and storing a first validity period of the stable area and a second validity period of the unstable area in the target map.

[0088] The first validity period is greater than the second validity period.

[0089] The embodiment of the present application does not limit the location of storing the target map, and can be configured according to actual needs. For example, it can be stored in the vehicle local terminal, in the cloud, or at both ends at the same time.

[0090] Here, the target map can be stored in a designated location on the local vehicle side, or in a designated location in the cloud.

[0091] Stable areas are areas whose location and appearance do not change over a long period of time, such as residential areas.

[0092] Unstable areas are areas whose location and appearance may change over time, such as construction areas.

[0093] The embodiment of the present application does not limit the duration of the first validity period and the second validity period, and can be configured according to actual needs. For example, the second validity period can be configured according to the construction period.

[0094] S105 may be implemented as follows: storing the target map at a designated location, and then determining the stable area and the unstable area in the target map based on the user's input operation. For the stable area in the target map, a first validity period is added based on the user's operation; for the unstable area, a second validity period is added based on the user's operation.

[0095] In this embodiment, the method includes: based on the image data and point cloud data of the vehicle during driving, respectively determining the target image features of multiple target objects and the target point cloud features of multiple target objects in the vehicle driving environment; based on the target image features of the multiple target objects and the first positions of the multiple target objects, constructing a three-dimensional visual map of the vehicle; the first position is obtained based on the image data; based on the target point cloud features of the multiple target objects and the second positions of the multiple target objects, constructing a three-dimensional laser map of the vehicle; the second position is obtained based on the point cloud data; and determining a target map for assisting vehicle driving based on at least the three-dimensional visual map and the three-dimensional laser map.

[0096] Based on the aforementioned technical approach, a 3D visual map is first constructed from image data, and a 3D laser map is constructed from point cloud data. The target map is then determined based on the 3D visual map and the 3D laser map. This combines the advantages of both 3D visual and 3D laser maps, improving the accuracy and application scenarios of the target map and enhancing vehicle safety. Furthermore, the validity period of different areas can be clearly displayed, enhancing the effectiveness of the target map display and improving the user experience.

[0097] Next, the process of determining the target map for assisting vehicle driving based on at least the three-dimensional visual map and the three-dimensional laser map in S104 will be described.

[0098] The process may include but is not limited to implementation 1 or implementation 2 described below.

[0099] Implementation 1. Determine the target map based on the 3D visual map and 3D laser map.

[0100] For example, based on different scenarios, one of the three-dimensional visual map and the three-dimensional laser map can be selected as the target map. For another example, the three-dimensional visual map and the three-dimensional laser map can be fused to obtain the target map.

[0101] Implementation 2: Determine the target local map based on the 3D visual map and the 3D laser map; determine the target map based on the target local map and the vehicle's global map.

[0102] Since image data and point cloud data are collected in real time and the detection distance of cameras and radar sensors is limited, the obtained three-dimensional visual maps and three-dimensional laser maps are both local maps. The global map can be obtained based on navigation data.

[0103] The target local map can be determined based on the three-dimensional visual map and the three-dimensional laser map, and then the corresponding area in the entire map can be updated based on the target local map to obtain the target map.

[0104] Based on the above technical means, when determining the target map, if the target map is determined directly based on the three-dimensional visual map and the three-dimensional laser map, the advantages of both can be combined to improve the accuracy of the local map; when determining the target map, if it is determined based on the vehicle's global map, three-dimensional visual map and three-dimensional laser map, not only the accuracy of the local map is improved, but also the global planning is improved, the comprehensiveness of the target map is improved, and the accuracy of driving is improved.

[0105] Next, the process of determining the target local map based on the three-dimensional visual map and the three-dimensional laser map in implementation 2 is described.

[0106] The process may include but is not limited to the following method 1 or method 2.

[0107] Method 1: Determine the target local map based on the switching mechanism;

[0108] Method 2: Determine the target local map based on the fusion mechanism.

[0109] Next, the process of determining the target local map based on the switching mechanism in method 1 is described.

[0110] When the first condition is met, the three-dimensional visual map is determined as the target local map; when the first condition is not met, the three-dimensional laser map is determined as the target local map; wherein the first condition is used to characterize that the light intensity in the vehicle driving environment meets the light intensity threshold.

[0111] In a driving environment that meets the light intensity threshold, the image data collected by the camera is clearer and the resulting three-dimensional visual map is more accurate.

[0112] The embodiment of the present application does not limit the value of the light intensity threshold and can be configured according to actual needs.

[0113] Based on the above technical means, the target local map is determined based on the light intensity in the vehicle's driving environment. When the light intensity meets the intensity threshold, the image information is more comprehensive and the resulting visual map is more accurate. Therefore, the target local map determined based on the three-dimensional visual map is also accurate. When the light intensity does not meet the intensity threshold, the image information is low, the resulting visual map has more loss and is less accurate. Therefore, the target local map is determined based on the three-dimensional laser map. The target map obtained in this way can meet the needs of various scenarios, improving accuracy and application scenarios.

[0114] Next, the process of determining the target local map based on the fusion mechanism in method 2 is described.

[0115] refer to Figure 2 The process may include but is not limited to the following S201 to S204.

[0116] S201: Determine a first position of each target object in the three-dimensional visual map.

[0117] S201 can be implemented as follows: determining the position of each target object based on the image data to obtain a first position of each target object. The method for determining the position based on the image in the embodiment of the present application is not limited and can be configured according to actual needs. For example, the first position of the target object can be determined based on images of the target object at different angles.

[0118] S202: Determine a second position of each target object in the three-dimensional laser map.

[0119] S202 may be implemented as follows: determining the position of each target object based on the point cloud data to obtain the second position of each target object. Since the point cloud information is distance information, the second position of the target object can be obtained by performing a reference position transformation on the point cloud data.

[0120] S203: Fusing the first position and the second position through a filter to obtain a target position of each target object.

[0121] The embodiment of the present application does not limit the type of filter and can be configured according to actual needs.

[0122] For example, the filter may be an extended Kalman filter.

[0123] S203 may be implemented as follows: inputting the first position and the second position into a filter, fusing the first position and the second position through the filter processing to obtain the target position of each target object. Here, a fusion weight of the first position and the second position may also be configured in the filter.

[0124] S204: Determine a target local map based on the target feature of each target object and the target position of each target object.

[0125] The target features are obtained based on image features and / or point cloud features.

[0126] The target feature can be an image feature, a point cloud feature, or a feature that is a fusion of image features and point cloud features.

[0127] When fusing image and point cloud features, you can assign different weights to image and point cloud features before performing feature fusion. This weight can be shared across all target objects, or it can be assigned to different objects. For example, if an object is obscured by a shadow and has less image feature data, you can assign a slightly higher weight to the point cloud features.

[0128] S204 can be implemented as follows: for each target object, the position of the target object in the target local map is located by the target position of the target object, and then at the target position, the target object is rendered based on the target feature of the target object, thereby forming the target local map.

[0129] Here you can configure the granularity of target object rendering according to actual needs.

[0130] Based on the above technical means, when determining the local map of the target, the three-dimensional visual map and the three-dimensional laser map can also be fused. Since the map generally includes objects and their positions, fusing the position of each object improves the accuracy of the position, and fusing the features improves the accuracy of the features.

[0131] It should be noted that the target map in implementation 1 can be determined by referring to the method of determining the target local map, which will not be described in detail here.

[0132] Next, the process of obtaining the target object is described.

[0133] In one possible implementation, reference Figure 3 The process may include but is not limited to the following S301 and S302.

[0134] S301: Identify image data to obtain at least one static object and at least one dynamic object.

[0135] S301 may be implemented as follows: using an image recognition algorithm to process the image data, identifying objects in the image data, and then using a classification model to determine the type of each object, thereby obtaining at least one static object and at least one dynamic object.

[0136] S302: Filter at least one identified dynamic object and determine at least one static object as a target object.

[0137] That is, the position and feature information of dynamic objects are filtered out.

[0138] S302 may be implemented as follows: filtering the at least one identified dynamic object, filtering out the position information and feature information of the dynamic object, and determining the at least one static object after filtering as the target object.

[0139] Based on the above technical means, dynamic objects are filtered out. Since dynamic objects have dynamic errors, they will affect the accuracy of the map. The three-dimensional map determined based on static objects is more accurate.

[0140] In another possible implementation, both static objects and dynamic objects may be determined as target objects.

[0141] Next, the implementation process of constructing the three-dimensional visual map of the vehicle based on the target image features of the multiple target objects and the first positions of the multiple target objects in 102 is described.

[0142] In a possible implementation manner, when the target object includes a static object and a dynamic object, reference Figure 4 The process may include but is not limited to the following S401 and S402.

[0143] S401: Determine a first influence coefficient of a static object and a second influence coefficient of a dynamic object respectively.

[0144] The first influence coefficient is greater than the second influence coefficient.

[0145] The embodiment of the present application does not limit the values ​​of the first influence coefficient and the second influence coefficient, and can be configured according to actual needs.

[0146] S401 can be implemented as follows: determining the first influence coefficient of the static object and the second influence coefficient of the dynamic object respectively according to the user input; or determining the first influence coefficient of the static object and the second influence coefficient of the dynamic object respectively based on the influence of the static object and the dynamic object on the result.

[0147] S402: Construct a three-dimensional visual map based on the target image features, first positions, and first influence coefficients of static objects and the target image features, first positions, and second influence coefficients of dynamic objects.

[0148] S402 can be implemented as follows: for each static object, based on the first influence coefficient, the first position and the target image feature, the display of the static object in the three-dimensional visual map is determined; for each dynamic object, based on the second influence coefficient, the first position and the target image feature, the display of the dynamic object in the three-dimensional visual map is determined.

[0149] Based on the above technical means, when the target objects include static objects and dynamic objects, by configuring the first influence coefficient and the second influence coefficient, the influence of static objects on the three-dimensional map is increased, and the influence of dynamic objects on the three-dimensional map is reduced, and the obtained three-dimensional map is more accurate.

[0150] In another possible implementation, when the target objects include only static objects, the three-dimensional visual map is determined based only on dynamic objects.

[0151] Next, the process of determining target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle driving environment based on the image data and point cloud data of the vehicle during driving in S101 will be described.

[0152] The process of determining image features is similar to that of determining point cloud features. The following uses image features as an example to illustrate the process. The process of determining point cloud features can refer to the description of the image feature process.

[0153] refer to Figure 5 The process may include but is not limited to the following S501 to S503.

[0154] S501: Identify image data of a moving vehicle using a first network model to obtain initial feature maps of multiple target objects.

[0155] The embodiment of the present application does not limit the type of the first network model, and can be configured according to actual needs. For example, the first network model can be a multi-target detection model.

[0156] S501 can be implemented as follows: inputting an image of a moving vehicle into a first network model, performing recognition processing on the image data of the moving vehicle through the first network model, and obtaining initial feature maps of multiple target objects.

[0157] S502: Perform target-scale pooling processing on the initial feature maps of the multiple target objects through the second network model to obtain target-scale feature maps.

[0158] For example, the second network model may be DeepLab v3. For example, the second network model may include: a backbone network and a spatial pyramid pooling module.

[0159] The embodiment of the present application does not limit the size of the target scale, and can be configured according to actual needs. The target scale here can be one scale or multiple scales.

[0160] S502 may be implemented as follows: inputting the initial feature maps of the plurality of target objects into the second network model, and performing target-scale pooling processing on the initial feature maps of the plurality of target objects by the second network model to obtain feature maps of the target scale. The target scale is pre-configured in the second network model.

[0161] S503: Concatenate the target scale feature map and the initial feature map to obtain target image features of multiple target objects.

[0162] S503 can be implemented as follows: performing feature map splicing processing on the feature map of the target scale and the initial feature map to obtain target image features of multiple target objects. If the target scale is multiple scales, the feature maps of the multiple scales are spliced ​​with the initial feature map respectively.

[0163] Based on the above technical means, when determining the target image features, the target scale can be configured based on actual needs, so that the target image features that meet the actual needs can be obtained.

[0164] Among them, the target scale can be configured according to actual needs. When the target scale includes multiple scales, the target image features include target image features at multiple scales; correspondingly, the three-dimensional visual map includes three-dimensional visual maps of multiple resolutions; and the target map includes target maps of multiple resolutions.

[0165] refer to Figure 6 The map determination method provided in this embodiment may further include but is not limited to the following S601 and S602.

[0166] S601: When a second condition is met, assist vehicle driving based on a target map with a first resolution.

[0167] The second condition is used to characterize the vehicle's driving environment as an open environment. The embodiments of the present application do not limit the method for determining an open environment, and can be configured according to actual needs. The determination can be based on image data; for example, if grasslands, plains, or other scenes are detected, the environment is determined to be an open environment. The determination can also be based on point cloud data; for example, if the average distance of the detected objects is greater than a distance threshold, the vehicle is determined to be in an open environment.

[0168] The embodiment of the present application does not limit the value of the first resolution and can be configured according to actual needs.

[0169] S601 may be implemented as follows: when the second condition is met, calling the target map of the first resolution and outputting the target map of the first resolution to assist the vehicle in driving. In this way, system resources can be saved.

[0170] S602: When the third condition is met, assist the vehicle in driving based on the target map with the second resolution.

[0171] The first resolution is lower than the second resolution.

[0172] The third condition is used to characterize that the driving environment of the vehicle is a narrow environment or a complex environment.

[0173] The embodiments of this application do not limit the method for determining an open environment and can be configured according to actual needs. The determination can be based on image data; for example, if various tall buildings, pedestrians, traffic lights, and other scenes are detected, the environment can be determined to be narrow or complex. The determination can also be based on point cloud data; for example, if the average distance of the detected objects is less than a distance threshold, the environment can be determined to be narrow or complex.

[0174] The embodiment of the present application does not limit the value of the second resolution and can be configured according to actual needs.

[0175] In simple terms, the second resolution may also be referred to as high resolution, and the first resolution may also be referred to as low resolution.

[0176] S602 may be implemented as follows: when the third condition is met, calling the target map of the second resolution and outputting the target map of the second resolution to assist the vehicle in driving. In this way, the needs of actual scenarios can be met.

[0177] Based on the above technical means, target maps with multiple resolutions can be obtained. Target maps with different resolutions can be switched in different driving scenarios to meet the needs of various scenarios.

[0178] The map determination method provided in the embodiment of the present application can also process image data and point cloud data, and determine image features and point cloud features based on the processed image data and point cloud data.

[0179] refer to Figure 7 The process may include but is not limited to the following S701 to S705.

[0180] S701: Acquire initial image data and initial point cloud data of a moving vehicle.

[0181] The initial image data herein refers to image data captured by a camera. The embodiments of the present application do not limit the camera used to capture the initial image data, and the camera may be configured according to actual needs. For example, the camera may be located in front of the vehicle. Alternatively, the camera may be located in all cameras on the vehicle used to capture external images.

[0182] The initial point cloud data here refers to point cloud data collected by radar. The embodiments of this application do not limit the radar used to collect the initial point cloud data, and can be configured based on actual needs. For example, it can be a millimeter-wave radar and / or a lidar. The radar's deployment location can also be configured and selected based on actual needs.

[0183] S701 can be implemented as follows: collecting initial image data and initial point cloud data of the vehicle while driving; or storing the initial image data and initial point cloud data while driving after collection, and directly calling the stored initial image data and initial point cloud data here.

[0184] S702 : Perform filtering processing on the initial image data and the initial point cloud data to obtain first image data and first point cloud data.

[0185] The embodiment of the present application does not limit the filtering algorithm and can be configured according to actual needs.

[0186] The filtering method may include but is not limited to one or more of the following: low-pass filtering, Kalman filtering, mean filtering, median filtering, extended Kalman filtering, and unscented Kalman filtering.

[0187] S702 may be implemented as follows: the electronic device performs filtering processing on the initial image data and the initial point cloud data respectively through one or more of the above filtering methods to obtain first image data and first point cloud data.

[0188] S703 : Detect and correct abnormal data on the first image data and the first point cloud data to obtain corrected second image data and second point cloud data.

[0189] The embodiments of the present application do not limit the detection and correction algorithms for abnormal data and can be configured according to actual needs.

[0190] Abnormal data detection can be achieved through threshold detection and time series analysis. For example, by setting a reasonable threshold range, when sensor data exceeds the normal range, the system will mark it as abnormal data and trigger the correction process. Using time series analysis methods, including algorithms such as moving averages and autoregressive models, by analyzing the historical trend of data and identifying anomalies, it can effectively detect short-term mutations and long-term drift data, thereby triggering corresponding correction strategies.

[0191] Detecting abnormal data can also be achieved through statistical anomaly detection algorithms. For example, based on the 3σ rule of mean and variance, the mean and standard deviation of collected sensor data are calculated. If a data point deviates from the mean by more than three times the standard deviation, the data is considered abnormal, improving the robustness of the system.

[0192] Data correction can be achieved by resampling abnormal data points and obtaining new sensor data to replace the abnormal points.

[0193] For data correction, linear interpolation or spline interpolation can also be used to generate new data by surrounding data points and replace outliers.

[0194] Data correction can also be achieved by using historical data for retrospective analysis and correcting current data through weighted average or prediction models.

[0195] S704 : Time-align the second image data and the second point cloud data using a time interpolation technique to obtain aligned third image data and third point cloud data.

[0196] The embodiment of the present application does not specifically limit the method of time alignment and can be configured according to actual needs.

[0197] For example, the data collected by each sensor is accompanied by timestamp information, and the data is synchronized through the timestamp, and data with different sampling rates are converted to the same time axis; before multi-sensor data fusion, the system aligns the data. The data alignment uses timestamp interpolation technology to align the output of each sensor in a way that minimizes time delay.

[0198] To avoid state lags caused by data delays, a delay compensation mechanism can be introduced. This mechanism estimates data delays through a prediction model and uses the estimated values ​​to predict states. For example, when image data lags behind point cloud data, short-term predictions can be made based on the image data's changing trends to compensate for the impact of this delay. Delay optimization requires adaptive updates based on the driving environment and sensor characteristics. The system regularly monitors changes in data delays and dynamically adjusts delay compensation parameters to ensure real-time performance in various environments.

[0199] S705 : Based on the third image data and the third point cloud data of the vehicle during driving, respectively determine target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle driving environment.

[0200] The implementation of S705 is similar to that of S101 and will not be described in detail here.

[0201] Through the above data optimization and processing flow, the accuracy and real-time performance of multi-sensor fusion data can be ensured, providing data support for the establishment of accurate maps.

[0202] Based on the above technical means, the initial image data and initial point cloud data can be filtered, anomaly corrected, time-aligned, etc., so that the processed data can better meet the needs, improve the accuracy of the data, and thus improve the accuracy of the target map and driving safety.

[0203] The map determination method provided in the embodiment of the present application also includes a target map adjustment process.

[0204] The adjustment process may include but is not limited to the following time graph-based adjustment process and pose graph-based adjustment process.

[0205] refer to Figure 8 As shown in the content, the adjustment process based on the time diagram may include but is not limited to the following S801 to S804.

[0206] S801: Construct a first node based on the vehicle posture and environmental characteristics at each moment.

[0207] Environmental features are obtained based on image features and / or point cloud features.

[0208] The environmental features may be image features and / or point cloud features, or may be features fused together based on their respective weights.

[0209] S801 may be implemented as follows: constructing a first node based on a time graph construction algorithm for the vehicle posture and environmental features at each moment.

[0210] S802: Determine a first constraint relationship between adjacent first nodes as a matching degree between environmental features of the adjacent first nodes.

[0211] The matching degree between the environmental features of each two first nodes is determined, thereby obtaining a first constraint relationship between each two first nodes. Here, each two first nodes are first nodes corresponding to the time when they are connected.

[0212] S802 can be implemented as follows: based on the time graph construction algorithm, for adjacent first nodes, first determine the matching degree between the environmental features of the adjacent first nodes, and then add the first constraint relationship between the adjacent first nodes based on the matching degree between the environmental features of the adjacent first nodes.

[0213] S803: Construct a time graph based on the first nodes and the first constraint relationships between the first nodes.

[0214] A time graph is constructed based on the first nodes constructed at all moments and the first constraint associations between the first nodes corresponding to every two adjacent moments.

[0215] S803 can be implemented as follows: the electronic device traverses the vehicle posture and environmental characteristics of the vehicle at each moment, constructs all first nodes, traverses every two first nodes, and adds a first constraint relationship based on the matching degree of the environmental characteristics between every two first nodes, thereby obtaining a time graph.

[0216] S804: Adjust the target map by adjusting the first nodes in the time graph and the constraint relationships between the first nodes.

[0217] The positioning error is reduced by optimizing the positions of nodes and the constraints between edges.

[0218] S804 may be implemented as follows: adjusting the first nodes in the time graph and the constraint relationships between the first nodes to reduce the positioning error and optimize the target map.

[0219] Based on the above technical means, the target map can be adjusted. Adjustment based on the time map can reduce the time accumulation error, thereby improving the accuracy of the target map and driving safety.

[0220] refer to Figure 9As shown in the content, the adjustment process based on the pose graph may include but is not limited to the following S901 to S904.

[0221] S901: Construct a second node based on different positions of the vehicle.

[0222] A plurality of second nodes are constructed by determining a location where the vehicle travels.

[0223] S901 may be implemented as follows: based on a pose graph construction algorithm, a second node is constructed for each driving position of the vehicle, and different second nodes are constructed for different positions of the vehicle.

[0224] For example, the relative pose between the loop node and the history node is added as a new constraint to the pose graph, a new loop constraint is generated, and the position is marked.

[0225] S902: Determine the edge between every two second nodes as the posture change value between every two second nodes.

[0226] Every two second nodes correspond to two second nodes corresponding to two positions that are always continuous in the position change process.

[0227] An edge between each two second nodes is determined based on a pose change value between each two second nodes.

[0228] S902 can be implemented as follows: based on the pose graph construction algorithm, for adjacent second nodes, first determine the pose change value between every two second nodes, and then add the edge between adjacent second nodes based on the pose change value between every two second nodes.

[0229] The confirmed loop constraints can also be added to the pose graph to strengthen the relative position constraints between keyframes (a map of moments), adding a node for each keyframe and adding constraint edges between adjacent keyframes to reduce the cumulative error.

[0230] S903: Construct a pose graph based on the second node and the edge between the second node.

[0231] A pose graph is constructed based on all second nodes and the edges between every two second nodes.

[0232] S903 can be implemented as follows: the electronic device traverses each driving position of the vehicle to construct all second nodes, traverses the position change values ​​between all second nodes to construct edges between the second nodes, and obtains a posture graph.

[0233] S904: Adjust the second node and the edges between the second nodes in the pose graph, and adjust the target map.

[0234] S904 may be implemented as follows: using a graph optimization algorithm to adjust the second node and the edges between the second nodes in the pose graph by minimizing the error between the nodes, thereby achieving target map optimization.

[0235] Here, if the vehicle passes through the historical area, a new constraint is added after loop confirmation.

[0236] Based on the above technical means, the target map can be adjusted. Adjustments based on the pose graph can reduce the cumulative error caused by position changes, thereby improving the accuracy of the target map and driving safety.

[0237] The map determination method provided in the embodiment of the present application may also include updating the map.

[0238] refer to Figure 10 The updating process may include but is not limited to the following S106.

[0239] S106 : When it is detected that the second validity period of the unstable area has arrived, reconstruct and update the unstable area in the target map.

[0240] The embodiment of the present application does not limit the method of reconstructing and updating the map of the unstable area, and reference may be made to the description of the above creation process.

[0241] Based on the above technical means, the map information of unstable areas can be updated in a timely manner, achieving high precision of the target map while also having high real-time performance.

[0242] The map determination method provided in the embodiment of the present application may further include a prompting process for the predicted behavior of the target object.

[0243] refer to Figure 11 The process may include but is not limited to the following S1101 and S1102.

[0244] S1101. Determine predicted behaviors of multiple target objects.

[0245] The predicted behavior of the target object can also be predicted through behavior prediction algorithms.

[0246] For example, in complex traffic environments, there may be multiple dynamic traffic participants (such as pedestrians, bicycles, and other vehicles). A behavior prediction model can be used based on multi-target tracking technology to track the trajectories of different target objects and, combined with environmental semantic information, predict the future position of each target. This allows us to determine whether the target object's trajectory affects the vehicle's movement, thereby improving the system's overall predictive capabilities.

[0247] S1101 may be implemented as follows: predicting behaviors of multiple target objects through a prediction model or a prediction algorithm, thereby obtaining predicted behaviors of the multiple target objects.

[0248] S1102: If the predicted behavior of the target object affects the driving of the vehicle, output a first prompt message on the target map.

[0249] S1102 may be implemented as follows: matching the trajectory of each target object's predicted behavior with the vehicle's predicted trajectory to determine whether the target object's predicted behavior affects the vehicle's travel; if it is determined that the target object's predicted behavior affects the vehicle's travel, outputting a first prompt message on the target map. If it is determined that the target object's predicted behavior does not affect the vehicle's travel, no prompt message is provided.

[0250] The first prompt information is used to prompt that the predicted behavior of the target object affects the driving of the vehicle.

[0251] The embodiment of the present application does not limit the prompting method of the first prompting information, and can be configured according to actual needs. For example, the first prompting information can be a voice prompt. Alternatively, the target object that affects the vehicle's driving can be highlighted.

[0252] Based on the above technical means, behaviors that affect vehicle driving can be output in the target map, which improves the richness of the target map display and the safety of vehicle driving.

[0253] It should be noted that the descriptions of the above embodiments and steps can be combined according to actual needs if there is no contradiction, and they will not be described one by one here.

[0254] Below, taking an assisted driving vehicle as an example, the map determination process provided by this embodiment of the present application is described.

[0255] With the rapid development of autonomous driving and intelligent transportation technologies, precise positioning has become a key factor in ensuring the normal operation of intelligent vehicles in complex environments. Traditional positioning technologies, particularly those based on the Global Positioning System (GPS), can provide relatively accurate location information in open suburban areas or on highways, with an error typically between 1 and 3 meters. However, in complex urban environments (such as densely populated cities, tunnels, and underground garages), GPS positioning accuracy can be significantly reduced or even ineffective due to signal reflection, obstruction, and interference, making it unable to meet the high-precision positioning requirements of autonomous driving systems.

[0256] Furthermore, relying solely on a single sensor for vehicle positioning has many limitations in practical applications. For example, when GPS and an inertial measurement unit (IMU) are without signal for extended periods, the IMU sensor can easily accumulate errors, leading to positioning drift.

[0257] To address the shortcomings of single-positioning technologies, modern driver assistance systems are beginning to adopt multi-sensor fusion technology. This combines data from multiple sensors (such as GPS, IMU, LiDAR, and cameras) to improve positioning accuracy and reliability. However, efficiently integrating this data and using algorithms to process massive amounts of environmental information in real time remains a key technical challenge.

[0258] Therefore, this embodiment proposes a high-precision positioning solution that combines multi-sensor fusion technology, deep learning, and Simultaneous Localization and Mapping (SLAM) technology. This solution, capable of maintaining high-precision assisted driving positioning in a variety of complex environments, has become a pressing technical challenge. This embodiment addresses these shortcomings by providing an efficient, stable, and low-latency precision positioning solution for assisted driving systems.

[0259] The purpose of this embodiment is to develop an assisted driving precise positioning system suitable for complex road environments and diverse climatic conditions, to improve the positioning accuracy of existing autonomous driving vehicles in complex scenarios, especially in situations where the signal is unstable or vision is limited, to ensure that the vehicle can obtain high-precision location information in real time, thereby improving the accuracy and safety of the autonomous driving system's decision-making, planning, and control.

[0260] This embodiment provides a precise positioning method and system that integrates multi-sensor data fusion, deep learning algorithms, and SLAM technology. The main technical solution includes the following steps:

[0261] Step 1: Multi-sensor data acquisition.

[0262] The vehicle is equipped with multiple different types of sensors, including GPS, inertial measurement unit, lidar, millimeter-wave radar and camera, which are used to collect real-time location information, speed, acceleration, direction, environmental obstacles, etc.

[0263] Step 2: Data fusion and filtering processing.

[0264] Multi-sensor fusion technology is key to this implementation. By fusing data from various sensors, more accurate vehicle location information is obtained. Because different sensors have varying data acquisition frequencies, accuracies, and reliability, a filtering algorithm is used to achieve synchronous fusion of multi-source data.

[0265] Among them, the extended Kalman filter performs state estimation on nonlinear systems, solving nonlinear problems in sensor data and enabling real-time correction of drift errors in GPS and IMU data. The unscented Kalman filter performs nonlinear filtering on sensor data in complex scenarios, adapting to more complex dynamic environments and improving the accuracy and stability of data fusion through multiple iterations. The purpose of fusion processing is to integrate data from various sensors and compensate for potential flaws in individual sensors. For example, when the GPS signal is weak, pose estimation relies on data from the IMU and lidar. When the camera image is blurry, lidar and millimeter-wave radar detection data can serve as alternatives.

[0266] Step 3: Scene understanding and target detection.

[0267] Step 4: Real-time map construction and update based on SLAM.

[0268] To compensate for the shortcomings of GPS signals in complex environments, this embodiment introduces simultaneous localization and mapping (SLAM) technology. SLAM uses the surrounding environment data obtained by cameras and lidar to construct a local three-dimensional map of the vehicle in real time and match it with the pre-stored global map to ensure that the vehicle can still be accurately positioned based on the local map when the GPS signal is weak or fails.

[0269] Visual Simultaneous Localization and Mapping (VSLAM) and Laser-based Simultaneous Localization and Mapping (LSLAM) are combined. VSLAM uses visual information acquired by cameras for positioning, while LSLAM uses lidar to scan and obtain more accurate depth information and environmental contours. In practical applications, the combination of VSLAM and LSLAM enables the system to adapt to different environmental conditions and ensures reliable positioning.

[0270] Map matching and optimization: The local map constructed by SLAM technology will be matched with the global high-precision map in real time, and the continuity and accuracy of the map will be ensured through the graph optimization algorithm. Especially in environments where GPS signals fail, such as tunnels and underground parking lots, this method can effectively prevent positioning drift.

[0271] Step 5: Positioning error correction and optimization.

[0272] Through the fusion of multi-sensor data, the application of deep learning algorithms and the local map constructed by SLAM, the vehicle's positioning error can be corrected in real time to ensure that the error remains at the centimeter level; dynamic error correction, when the vehicle's position information drifts, the system will compare the real-time data obtained by the sensor with the map, and dynamically adjust the vehicle's position through the fusion processing algorithm to make it consistent with the actual road environment; intelligent optimization, the system combines deep learning models to perform intelligent analysis of the vehicle's surroundings, predict the movement trends of traffic participants, optimize driving paths and obstacle avoidance strategies, and further improve positioning accuracy.

[0273] Step 6: Output high-precision positioning information.

[0274] It has the following advantages:

[0275] 1. This embodiment integrates data from GPS, IMU, LiDAR, millimeter-wave radar, and cameras, achieving high-precision positioning through algorithms such as the Extended Kalman Filter (EKF) and the Unscented Kalman Filter (UKF). Furthermore, the weighting of each sensor is dynamically adjusted based on the real-time environment, for example, increasing the weight of LiDAR in low-light conditions and GPS in open areas. This adaptive weighting mechanism ensures accurate and robust positioning, effectively addressing various complex driving environments.

[0276] 2. This embodiment combines the advantages of VSLAM and LSLAM, dynamically switching and co-using them according to environmental conditions. By fusing visual and laser data, it overcomes the limitations of single SLAM methods in different environments, such as the instability of visual SLAM in poor lighting conditions and the feature loss problem of laser SLAM in open areas.

[0277] 3. When processing pedestrians, vehicles, and other moving objects in dynamic environments, a dynamic object filtering method combining semantic segmentation and optical flow detection is used to remove dynamic object feature points and point cloud data to prevent interference with the vehicle's precise positioning. At the same time, positioning errors are corrected using static feature points to ensure accurate positioning, ensuring stable operation even in dynamic environments such as congestion or complex intersections.

[0278] 4. The loop detection function can identify repeated passages of vehicles within a certain area. By adding new constraints at the loops, graph optimization is achieved, thereby reducing cumulative errors. Especially in the case of long-distance driving, this patent effectively corrects map deviations through loop detection and pose graph optimization, thereby improving the accuracy and consistency of overall map construction.

[0279] 5. This embodiment provides an efficient map caching and reuse mechanism. By locally storing and managing local maps, vehicles can directly access cached map data when passing through the same area multiple times, avoiding repeated calculations and significantly improving system efficiency. The system also supports uploading updated maps to the cloud, sharing the latest map data with other autonomous vehicles, enabling multi-vehicle collaborative updates and improving the timeliness and consistency of map data.

[0280] 6. In terms of scene understanding, this patent integrates You Only Look Once (YOLO) object detection, DeepLabv3+ semantic segmentation, and behavior prediction based on Long Short-Term Memory (LSTM) networks to build a comprehensive environmental perception system. This system not only detects static objects and lanes but also predicts the behavior of dynamic objects (such as the future trajectories of pedestrians and vehicles), providing more accurate decision support for autonomous driving systems and ensuring safety in complex scenarios.

[0281] 7. A multi-resolution map construction method is adopted to adapt to different driving environments and map accuracy requirements. Low-resolution maps are used in open scenes to save computing resources, while high-resolution maps are used in complex urban areas or intersections to obtain detailed environmental information. In addition, the LSTM network dynamically compensates for accumulated errors, reducing positioning drift during long-term operation and improving long-term positioning stability.

[0282] Next, the modules related to the data processing involved in this embodiment are described.

[0283] refer to Figure 12 The content shown includes: a data acquisition module 1201, a data processing module 1202, a scene understanding module 1203, a positioning and map construction module 1204, and a control and execution module 1205.

[0284] The data acquisition module 1201 is used to obtain all real-time information required by the vehicle during driving through multiple different types of sensors. The main sensors involved include GPS, IMU (inertial measurement unit), lidar, millimeter-wave radar and camera.

[0285] The data processing module 1202 is used to integrate data from GPS, IMU, lidar, millimeter-wave radar and camera.

[0286] The scene understanding module 1203 is used to process image data and point cloud data collected by sensors such as cameras and lidars, mainly through convolutional neural networks, to achieve real-time target detection and environment recognition.

[0287] The positioning and mapping module 1204 is used to provide high-precision positioning information even in environments where the GPS signal is weak or ineffective. This embodiment achieves accurate vehicle positioning and real-time map construction by introducing simultaneous positioning and mapping (SLAM) technology. SLAM technology can rely on lidar and cameras to generate a map of the vehicle's surroundings without a pre-known map, and synchronously update the vehicle's position.

[0288] After obtaining the vehicle's precise location, the control and execution module 1205 transmits this information to the vehicle's control module for path planning, decision-making, and control. The control module plans the optimal driving path based on the vehicle's current location and dynamic information about the surrounding environment, and executes corresponding operations by controlling the vehicle's steering, acceleration, and braking.

[0289] Next, the positioning process is described.

[0290] refer to Figure 13 The process may include but is not limited to the following S1301 to S1304.

[0291] S1301, data collection: including GPS, IMU, lidar, millimeter-wave radar, and camera data.

[0292] The system obtains real-time information about the vehicle and its surrounding environment from a variety of sensors. It achieves high-precision environmental perception and positioning by integrating multiple sensors such as GPS, IMU, lidar, millimeter-wave radar and cameras.

[0293] S1302, data preprocessing: noise filtering, abnormal data removal, time synchronization.

[0294] Data optimization processing ensures the accuracy and stability of multi-sensor data through multiple links such as fusion, denoising, anomaly detection and error correction; including noise filtering, multi-sensor data fusion, anomaly data detection and correction, dynamic weight adjustment, and data delay optimization.

[0295] S1303, Data fusion and filtering: Extended Kalman filter, Unscented Kalman filter, Weight adjustment.

[0296] Multi-sensor data fusion and filtering uses an improved Kalman filter and dynamic weighting algorithm to fuse multi-sensor data.

[0297] The extended Kalman filter can handle nonlinear systems and is suitable for data fusion of sensors such as GPS and IMU. In this embodiment, the extended Kalman filter is used to fuse GPS position data and IMU acceleration data, converting the sensor data into state estimates through a nonlinear function, thereby improving positioning accuracy.

[0298] Compared with the extended Kalman filter, the unscented Kalman filter is more suitable for data processing in complex scenarios, especially with better accuracy in high-dimensional nonlinear state spaces. The unscented Kalman filter fuses and optimizes data through unscented transformation, and is used for data fusion of lidar and millimeter-wave radar, making the system more stable in complex environments.

[0299] A dynamic weighted fusion mechanism has been introduced. Different sensors have different reliability under different conditions. For example, in a well-lit environment, the weight of camera data can be higher; in rainy and snowy weather, the weight of millimeter-wave radar data is higher. Through the weighted fusion system, the weight of each sensor data can be automatically adjusted according to the real-time environment to ensure the reliability of the output positioning data.

[0300] Dynamic weight adjustment,This embodiment proposes a dynamic weight adjustment strategy, which adjusts the weight of the sensor in real time according to the data quality and environmental conditions of the sensor to ensure the accuracy of the positioning data.

[0301] The weight assessment model is based on the real-time data quality, environmental conditions and historical data performance of the sensor. For example, in low-visibility weather, the weight of the camera will be relatively reduced, while the weight of the millimeter-wave radar and lidar will be increased.

[0302] The weights are adaptively adjusted. The system automatically adjusts the weights based on the confidence level of multi-sensor data. The higher the confidence level, the greater the weight. The confidence level is assessed based on the quality of the data collected in real time and the degree of deviation from the expected value. For example, when the GPS signal is stable, its weight is relatively high. If the signal fluctuates significantly, the weight is automatically reduced.

[0303] Based on dynamic weight adjustment, the system performs weighted fusion of multi-sensor data to ensure the accuracy of positioning results. The specific method is to perform linear weighted combination of data under different weights and obtain the optimal fused positioning data through optimization calculation.

[0304] S1304, data correction and output: dynamic object filtering, positioning error correction, and high-precision position information output.

[0305] Next, the process of data preprocessing, noise filtering, abnormal data removal, and time synchronization in S1302 is described.

[0306] Regarding noise filtering, this embodiment uses a variety of filtering technologies to perform noise reduction processing on the data to improve the reliability of the data.

[0307] Low-pass filters are used to remove high-frequency noise from sensor data. Kalman filters are used for real-time data processing. The state update process of Kalman filters includes two steps: prediction and update. Based on the vehicle's motion model, the current position and speed are predicted; the current sensor observations are used to correct the prediction results to obtain a high-precision state estimate.

[0308] Mean filtering is used to denoise lidar and camera data; median filtering is used to remove extreme outliers in lidar and millimeter-wave radar, especially when the vehicle is driving in rain, snow, or fog.

[0309] Abnormal data detection and correction. This embodiment uses multiple abnormality detection algorithms to identify and process abnormal data in real time, preventing erroneous data from affecting the system positioning accuracy.

[0310] Threshold-based detection and time series analysis methods, by setting a reasonable threshold range, when sensor data exceeds the normal range, the system will mark it as abnormal data and trigger the correction process; using time series analysis methods including algorithms such as moving average and autoregressive models, by analyzing the historical change trends of data and identifying anomalies, it can effectively detect short-term mutations and long-term drift data, thereby triggering corresponding correction strategies.

[0311] The statistical anomaly detection algorithm uses the 3σ rule of mean and variance to calculate the mean and standard deviation of the collected sensor data. If a data point deviates from the mean by more than 3 times the standard deviation, the data is considered an anomaly, thereby improving the robustness of the system.

[0312] Regarding the data correction strategy, after detecting abnormal data, the system will trigger the data correction mechanism. The specific methods include:

[0313] Resample the abnormal data points and obtain new sensor data to replace the abnormal points.

[0314] Use linear interpolation or spline interpolation to generate new data through surrounding data points to replace outliers.

[0315] Use historical data for retrospective analysis and revise current data through weighted average or forecasting models.

[0316] Data delay optimization: For this purpose, this embodiment designs a data delay optimization strategy to ensure that the data of all sensors are aligned at the same time, thereby ensuring the consistency of system processing.

[0317] Timestamp synchronization and data alignment: The data collected by each sensor is accompanied by timestamp information. The data is synchronized through timestamps, and data with different sampling rates are converted to the same time axis. Before multi-sensor data fusion, the system aligns the data. Data alignment uses timestamp interpolation technology to align the outputs of each sensor in a way that minimizes time delay.

[0318] Delay compensation and adaptive updates: To avoid state lags caused by data delays, the system introduces a delay compensation mechanism. This mechanism estimates data delays through a prediction model and uses the estimated values ​​to predict state. For example, if IMU data lags behind GPS data, short-term predictions can be made based on the changing trends of the IMU data to compensate for the impact of this delay. Delay optimization requires adaptive updates based on the driving environment and sensor characteristics. The system regularly monitors changes in data delays and dynamically adjusts delay compensation parameters to ensure real-time performance in various environments.

[0319] Through the above data optimization and processing flow, this embodiment can ensure the accuracy and real-time performance of multi-sensor fusion data, thereby maintaining high-precision positioning in complex autonomous driving environments, effectively responding to challenges such as severe weather and weak signal areas, and providing important technical support for the safety and stability of autonomous driving.

[0320] Next, the process of scene understanding is explained.

[0321] refer to Figure 14 The content shown may include but is not limited to the following S1401 to S1405.

[0322] S1401: Input camera image and radar point cloud data.

[0323] S1402, Object Detection: YOLO object detection, small object detection optimization, multi-object classification and bounding box.

[0324] Based on the YOLO algorithm, it detects lanes, pedestrians, vehicles, traffic signs and other targets in real time. The algorithm can simultaneously complete target positioning and classification in a single forward propagation. It has the advantages of high computational efficiency and fast detection speed, and is particularly suitable for autonomous driving scenarios.

[0325] The YOLO algorithm divides an image into multiple grids, with each grid responsible for detecting objects within that area. Each grid output contains the object's category probability, bounding box coordinates, and confidence score. The main process includes: scaling and normalizing the camera-captured image to fit the neural network's input requirements; dividing the image into a grid structure, with each grid predicting the category and location of one or more objects; predicting the bounding box coordinates, width, height, and confidence score for each object; and removing redundant bounding boxes through non-maximum suppression to retain the optimal detection results.

[0326] Multi-target detection and classification: In complex traffic environments, vehicles need to simultaneously identify multiple targets (such as pedestrians, vehicles, and traffic signs). This implementation uses an improved YOLOv4 model, which offers enhanced multi-target detection capabilities. YOLOv4 improves the Cross-Stage Partial Darknet53 (CSPDarknet53) architecture for feature extraction, improving model accuracy and speed through the use of pyramid pooling modules and cross-level connections.

[0327] Small target detection optimization: For distant or small targets, this embodiment uses a feature pyramid network (FPN) architecture to effectively improve the detection rate of small targets. By adding multi-scale feature maps during the feature extraction stage, FPN enhances the network's detection capabilities for small targets, resulting in higher accuracy in pedestrian detection and long-distance vehicle detection.

[0328] S1403, Semantic Segmentation: DeepLabv3+ segmentation lane detection, road, pedestrian, and obstacle segmentation.

[0329] Semantic segmentation classifies each pixel in a scene, assigning a specific category label, such as lane, road, pedestrian, or traffic sign. Semantic segmentation plays a crucial role in scene understanding, helping autonomous vehicles fully perceive road structure and environmental information. This implementation uses the deep learning-based semantic segmentation model, DeepLabv3+, combining feature extraction and multi-scale analysis to generate accurate pixel-level classifications.

[0330] The DeepLabv3+ model consists of a backbone network and an Atrous Spatial Pyramid Pooling (ASPP) module. The ASPP module pools multi-scale features to obtain rich spatial information, enabling the model to recognize objects at different scales. The specific process is as follows: Feature extraction, using the backbone network to extract convolutional features from the input image; Multi-scale pooling, where the ASPP module pools feature maps of different scales to generate feature maps with multi-scale information; Upsampling and concatenation, where the multi-scale feature maps are upsampled and concatenated with the initial feature map to form a high-resolution prediction result.

[0331] In the output stage of the semantic segmentation model, all pixels are classified into specific categories (such as roads, lanes, obstacles, etc.). Through the pixel-level classification results of semantic segmentation, the system can fully understand the spatial layout of roads and driving environments.

[0332] A specific lane line detection module is added to DeepLabv3+. Through special training, the model can effectively distinguish lane lines in multi-lane environments, ensuring that the vehicle can accurately identify the driving lane and improve driving safety.

[0333] S1404, behavior prediction: dynamic target tracking, pedestrian / vehicle trajectory prediction, multi-target behavior prediction.

[0334] S1405. Scene understanding result output: target location, classified road and obstacle information, and predicted dynamic behavior.

[0335] Behavior prediction and scenario result output. After completing basic target detection and semantic segmentation, this embodiment further introduces a scenario analysis and behavior prediction module to predict the behavior of other traffic participants and ensure that the vehicle can make reasonable predictions and avoidance.

[0336] By combining the results of target detection and semantic segmentation, a complete scene model is constructed. For example, after identifying a pedestrian, the scene analysis module can determine whether the pedestrian is on the zebra crossing, whether he or she is crossing the road, and other information, thereby inferring potential risks.

[0337] In addition to pedestrian and vehicle detection, a behavior prediction model has been added. Using a recurrent neural network (RNN) and a long short-term memory (LSTM) network, the system predicts the behavior of detected targets and infers their likely movement trajectory. For example, if a pedestrian is detected approaching a zebra crossing, the behavior prediction model can analyze their historical movement trajectory and predict whether they intend to cross the road.

[0338] In complex traffic environments, there may be multiple dynamic traffic participants (such as pedestrians, bicycles, and other vehicles). The behavior prediction model in this embodiment is based on multi-target tracking technology. By tracking the trajectories of different targets and combining them with environmental semantic information, it predicts the future location of each target, thereby improving the system's overall prediction capabilities.

[0339] Next, the positioning and map building process is explained.

[0340] The positioning and mapping of this embodiment combines visual SLAM (VSLAM), laser SLAM (LSLAM), map optimization and error correction technology to achieve high-precision, real-time positioning and mapping functions, combining the advantages of VSLAM and LSLAM to improve the system's positioning accuracy and environmental adaptability.

[0341] refer to Figure 15The process may include but is not limited to the following S1501 to S1506.

[0342] S1501: Input camera images and radar point cloud data.

[0343] S1502: Feature extraction and matching.

[0344] Feature points are extracted from images through algorithms such as Oriented FAST and Rotated BRIEF (ORB) and Scale Invariant Feature Transform (SIFT) to capture the boundaries of lane lines, obstacles, and other environmental objects.

[0345] Feature matching: matching feature points in adjacent frame images to determine the vehicle's moving posture in the environment.

[0346] Pose estimation, based on the matched feature points, uses the Perspectiven Point (PnP) algorithm to calculate the vehicle's pose changes and estimate the current position.

[0347] S1503, VSLAM module: visual feature point positioning, keyframe management, and 3D reconstruction.

[0348] VSLAM performs positioning based on the visual features of the camera, manages key frames, and reconstructs the three-dimensional environment to generate a preliminary local three-dimensional map.

[0349] S1504, LSLAM module: point cloud matching and alignment, laser point cloud map construction.

[0350] Feature extraction, point cloud matching, and map construction: By collecting point cloud data of the vehicle's surroundings, a three-dimensional point cloud reflecting the positions of obstacles around the vehicle is generated. Geometric features such as lines and surfaces are extracted from the point cloud for subsequent matching and positioning. The iterative closest point (ICP) algorithm is used to align the current frame's point cloud with the previous frame's point cloud to calculate the vehicle's posture transformation. The local map is updated in real time using the matched and aligned point cloud data to generate a more accurate three-dimensional environment model.

[0351] S1505, Data Fusion and Optimization: pose graph optimization, loop detection and correction, high-precision map update.

[0352] Data fusion and optimization of VSLAM and LSLAM. Using VSLAM or LSLAM alone has its limitations. VSLAM has difficulty providing stable pose estimation in environments with insufficient light or lack of texture (such as tunnels and at night), while LSLAM may lose environmental features in open spaces (such as highways). Therefore, this embodiment adopts a method of combining VSLAM and LSLAM, adaptively switching or using them together according to environmental conditions.

[0353] S1506. Positioning and map output: real-time high-precision location information and updated local map.

[0354] The switching mechanism uses the environmental perception module to monitor ambient lighting and feature richness in real time, adaptively switching SLAM modes under different conditions: In good lighting and rich textures, VSLAM is prioritized for positioning and mapping, fully utilizing visual data. In poor lighting and textures, the system switches to LSLAM mode, using lidar data to maintain positioning stability.

[0355] Fusion mechanism: In specific environments (such as urban blocks), the system can simultaneously utilize VSLAM and LSLAM data to obtain more accurate pose estimation through weighted fusion: feature point fusion, matching visual feature points with lidar point clouds to generate high-precision local maps; data fusion algorithm, using extended Kalman filtering or particle filtering (PF) to fuse the pose estimation results of VSLAM and LSLAM to reduce positioning errors.

[0356] Next, the map optimization process is explained.

[0357] To ensure positioning accuracy and dynamic updating of the environment, this embodiment uses graph optimization technology to optimize the map. Graph optimization mainly represents the vehicle posture and environmental features at each moment as nodes in a graph structure. The constraint relationships between nodes (such as the matching relationship between feature points in adjacent frames) constitute the edges of the graph, and positioning errors are reduced by optimizing the relationship between nodes and edges.

[0358] This implementation introduces loop closure technology, which detects when a vehicle passes through previously visited areas and reduces accumulated error. When a loop is detected, the system adds new constraints to further optimize the map structure and reduce drift. In the SLAM backend, a pose graph is used for global optimization. Nodes in the pose graph represent different vehicle positions, and edges represent pose transformations between frames. Global map optimization is achieved by minimizing inter-node error.

[0359] refer to Figure 16The process may include but is not limited to the following S1601 to S1605.

[0360] S1601. Similarity detection: feature point matching and historical key frame comparison.

[0361] Similarity detection, judges the similarity between the current frame and the historical frames through feature matching and determines the loop position.

[0362] S1602, loop confirmation: determine the loop position and generate new constraints.

[0363] Loop confirmation, when the similarity detection is successful, the system confirms the loop position, adds the relative pose between the loop node and the history node as a new constraint to the pose graph, generates a new loop constraint, and marks the position.

[0364] S1603, constraint addition: add loop constraints and update the pose graph.

[0365] The confirmed loop constraints are added to the pose graph to strengthen the relative position constraints between keyframes, adding a node for each keyframe and adding constraint edges between adjacent keyframes to reduce the cumulative error.

[0366] S1604, global optimization: pose graph optimization, nonlinear error minimization, cumulative error correction.

[0367] Global optimization: After loop closure detection is completed, all nodes are re-optimized using a graph optimization algorithm. The relative pose error between adjacent frames is calculated based on the matched feature points to eliminate the accumulated error.

[0368] The pose graph is nonlinearly optimized using the Levenberg-Marquardt algorithm, and the globally optimized pose and map are finally obtained.

[0369] S1605, Map update and output: High-precision map update, positioning error correction.

[0370] Optimized high-precision map output, including corrected positioning errors, supports vehicle navigation and environmental perception.

[0371] In a dynamic environment, error correction is a key step to improve the positioning accuracy of the system. Errors in a dynamic environment may come from interference from pedestrians, vehicles or other dynamic objects. Therefore, this embodiment introduces multiple error correction strategies.

[0372] Dynamic object filtering: Use convolutional neural networks or optical flow methods to detect dynamic objects in the environment and filter their point cloud or image features to avoid interference with positioning. For example, semantic segmentation can be used to separate pedestrians and vehicles in an image and the feature points of these dynamic objects can be removed during feature matching.

[0373] Local map updates: Obstacles in dynamic environments may change over time, so the system incorporates a real-time update mechanism into the local map construction process. When environmental changes are detected (such as the vehicle ahead leaving the field of view), the system automatically updates the local map to remove outdated environmental features.

[0374] Adaptive Weight Adjustment: During map optimization, this embodiment uses an adaptive weight adjustment algorithm to adjust the weights of feature points based on their stability. For example, static features that remain unchanged for a long time (such as the corners of a building) are given higher weights, while dynamic features that appear briefly are weighted down or even removed, thereby improving overall map stability.

[0375] In SLAM systems, positioning accuracy is crucial to the safety of autonomous driving. This embodiment further improves positioning accuracy through the following strategies: high-precision positioning data fusion. This embodiment combines high-precision positioning data from the Global Navigation Satellite System (GNSS) and fuses the GNSS's absolute position information with the SLAM's relative pose estimation through an extended Kalman filter, improving the system's overall positioning accuracy. Multi-resolution maps. To adapt to different road environments, this embodiment constructs a multi-resolution map hierarchy. In open environments, low-resolution maps are used to reduce computational complexity. In narrow urban areas or complex intersections, high-resolution maps are used to obtain more detailed environmental information. Long-short-term memory error compensation. This embodiment dynamically compensates for SLAM errors through an LSTM model. LSTM can learn the vehicle's error patterns in specific environments, predict the current positioning error based on historical data, and compensate for it, reducing the system's accumulated error.

[0376] Next, the map storage process is explained.

[0377] In order to improve the efficiency and adaptability of the system, this embodiment designs a map storage and reuse mechanism, which is particularly suitable for scenarios where vehicles pass through the same area multiple times.

[0378] refer to Figure 17 The process may include but is not limited to the following S1701 to S1706.

[0379] S1701. Map data generation: VSLAM / LSLAM constructs and generates a local high-precision map.

[0380] S1702, map cache management: store the local map locally, set the cache validity period, and determine the map reuse conditions.

[0381] Local map cache: During vehicle driving, the system will cache the generated local map in local memory; when the vehicle passes through the same area again, it can directly use the cached map data to reduce repeated calculations.

[0382] S1703, map reuse: Detect when the vehicle enters the cache area again, and directly call the cached map data to reduce repeated calculations.

[0383] S1704, Cloud Sharing and Collaboration: Upload updated maps to the cloud, share map data with other vehicles, and receive the latest maps from the cloud.

[0384] When multiple autonomous vehicles are driving in the same area, the system supports uploading map data to the cloud, enabling map sharing and collaborative updates among multiple vehicles, further improving map accuracy and update efficiency.

[0385] S1705. Output the latest high-precision map data containing dynamic updates and collaborative information.

[0386] After obtaining the vehicle's precise location, the system transmits this information to the vehicle's control module for path planning, decision-making, and control. Based on the vehicle's current location and dynamic information about the surrounding environment, the control module plans the optimal driving path and executes corresponding operations by controlling the vehicle's steering, acceleration, and braking.

[0387] S1706, map update: Use static feature point correction to eliminate interference from dynamic objects.

[0388] The system regularly updates local maps based on their timeliness and accuracy. For frequently changing areas (such as road construction), the system automatically marks the expiration date of the map and rebuilds a new map upon expiration. For stable areas (such as residential areas), the map data is stored permanently for reuse.

[0389] In summary, the positioning and mapping module of this embodiment combines VSLAM, LSLAM, map optimization, dynamic error correction, high-precision data fusion, and map storage reuse technologies to provide an efficient and reliable positioning and environmental perception system for autonomous vehicles. This invention can adapt to complex and changing driving environments, ensuring that the vehicle maintains high-precision positioning capabilities in dynamic environments, significantly improving the safety and stability of autonomous driving systems.

[0390] In a second aspect, an embodiment of the present application provides a map determination device. The map determination device can be deployed in a vehicle. Figure 18As shown in the content, the map determination device 180 may include but is not limited to: a first determination 1801 , a first construction unit 1802 , a second construction unit 1803 , a second determination unit 1804 and a storage unit 1805 .

[0391] The first determining unit 1801 is configured to determine target image features of a plurality of target objects and target point cloud features of a plurality of target objects in a driving environment of the vehicle based on image data and point cloud data of the vehicle during driving;

[0392] A first constructing unit 1802 is configured to construct a three-dimensional visual map of the vehicle based on target image features of the plurality of target objects and first positions of the plurality of target objects; the first positions are obtained based on the image data;

[0393] A second construction unit 1803 is configured to construct a three-dimensional laser map of the vehicle based on target point cloud features of the plurality of target objects and second positions of the plurality of target objects; the second positions are obtained based on the point cloud data;

[0394] A second determining unit 1804 is configured to determine a target map for assisting vehicle driving based on at least the 3D visual map and the 3D laser map;

[0395] The storage unit 1805 is used to store the target map, determine the stable area and the unstable area in the target map, and store the first validity period of the stable area and the second validity period of the unstable area in the target map; the first validity period is greater than the second validity period.

[0396] In some embodiments, the second determining unit 1804 is further configured to:

[0397] The target map is determined based on the three-dimensional visual map and the three-dimensional laser map; or the target local map is determined based on the three-dimensional visual map and the three-dimensional laser map; the target map is determined based on the target local map and the global map of the vehicle.

[0398] In some embodiments, the second determination unit 1804 is also used to: determine the three-dimensional visual map as the target local map when the first condition is met; and determine the three-dimensional laser map as the target local map when the first condition is not met; wherein the first condition is used to characterize that the light intensity in the vehicle driving environment meets the light intensity threshold.

[0399] In some embodiments, the second determination unit 1804 is also used to: determine the first position of each target object in the three-dimensional visual map; determine the second position of each target object in the three-dimensional laser map; fuse the first position and the second position through a filter to obtain the target position of each target object; determine the target local map based on the target features of each target object and the target position of each target object; wherein the target features are obtained based on image features and / or point cloud features.

[0400] In some embodiments, the map determination device 180 may further include a recognition unit, which is used to: recognize image data to obtain at least one static object and at least one dynamic object; filter the recognized at least one dynamic object and determine at least one static object as a target object.

[0401] In some embodiments, when the target object includes a static object and a dynamic object, the first construction unit 1802 is also used to: determine the first influence coefficient of the static object and the second influence coefficient of the dynamic object respectively; the first influence coefficient is greater than the second influence coefficient; and construct a three-dimensional visual map based on the target image features, first position and first influence coefficient of the static object, and the target image features, first position and second influence coefficient of the dynamic object.

[0402] In some embodiments, the first determination unit 1801 is further used to: identify image data of a moving vehicle through a first network model to obtain initial feature maps of multiple target objects; perform target scale pooling processing on the initial feature maps of multiple target objects through a second network model to obtain feature maps of the target scale; and splice the feature maps of the target scale with the initial feature maps to obtain target image features of multiple target objects.

[0403] In some embodiments, the map determination device 180 may further include a processing unit, the processing unit being configured to, when the target scale includes multiple scales, the target image feature includes target image features at multiple scales; correspondingly, the three-dimensional visual map includes three-dimensional visual maps at multiple resolutions; and when the target map includes target maps at multiple resolutions, perform the following:

[0404] When the second condition is met, the vehicle is assisted in driving based on the target map of the first resolution; when the third condition is met, the vehicle is assisted in driving based on the target map of the second resolution; wherein the first resolution is lower than the second resolution, the second condition is used to characterize that the vehicle's driving environment is an open environment; the third condition is used to characterize that the vehicle's driving environment is a narrow environment or a complex environment.

[0405] In some embodiments, the first determining unit 1801 is further configured to:

[0406] Acquire initial image data and initial point cloud data of a moving vehicle; perform filtering processing on the initial image data and initial point cloud data to obtain first image data and first point cloud data; perform abnormal data detection and correction on the first image data and the first point cloud data to obtain corrected second image data and second point cloud data; perform time alignment on the second image data and the second point cloud data using a time interpolation technique to obtain aligned third image data and third point cloud data; and determine target image features of multiple target objects and target point cloud features of multiple target objects in a moving vehicle environment based on the third image data and the third point cloud data of the moving vehicle.

[0407] In some embodiments, the map determination device 180 may further include an adjustment unit configured to:

[0408] A first node is constructed based on the vehicle posture and environmental features of the vehicle at each moment; the environmental features are obtained based on image features and / or point cloud features; a first constraint relationship between adjacent first nodes is determined as a matching degree between environmental features of adjacent first nodes; a time graph is constructed based on the first nodes and the first constraint relationships between the first nodes; and a target map is adjusted by adjusting the first nodes and the constraint relationships between the first nodes in the time graph.

[0409] In some embodiments, the adjustment unit is also used to: construct a second node based on different positions of the vehicle; determine the edge between each two second nodes as the posture change value between each two second nodes; construct a pose graph based on the edge between the second node and the second node; adjust the second node and the edge between the second nodes in the pose graph, and adjust the target map.

[0410] In some embodiments, the storage unit 1805 is further configured to:

[0411] When it is detected that the second validity period of the unstable region has reached, the unstable region in the target map is reconstructed and updated.

[0412] In some embodiments, the map determination device 180 may further include a reminder unit configured to:

[0413] Determine the predicted behavior of multiple target objects; if the predicted behavior of the target object affects the driving of the vehicle, output a first prompt information on the target map; the first prompt information is used to prompt that the predicted behavior of the target object affects the driving of the vehicle.

[0414] In a third aspect, the present application further provides a vehicle, comprising a processor and a memory, wherein the memory stores a computer program or instructions, and when the computer program or instructions are executed by the processor, the method provided in the first aspect is implemented.

[0415] In a fourth aspect, an embodiment of the present application provides a storage medium, that is, a computer-readable storage medium, which stores a computer program or instructions. When the computer program or instructions are executed by a processor, the method provided in the first aspect above is implemented.

[0416] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the method provided in the first aspect is implemented.

[0417] It should be noted that the descriptions of the above storage medium, device, and program product embodiments are similar to the descriptions of the above method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, device, apparatus, and program product embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0418] It should be understood that “one embodiment” or “an embodiment” mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, “in one embodiment” or “in some embodiments” appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0419] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0420] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0421] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0422] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0423] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0424] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0425] The above description is only an implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for determining a map, characterized in that: The method comprises: Based on the image data and the point cloud data of the vehicle during driving, target image features of a plurality of target objects and target point cloud features of a plurality of target objects in the vehicle driving environment are determined respectively; constructing a three-dimensional visual map of the vehicle based on target image features of the plurality of target objects and first positions of the plurality of target objects, wherein the first positions are obtained based on the image data; constructing a three-dimensional laser map of the vehicle based on target point cloud features of the multiple target objects and second positions of the multiple target objects, wherein the second positions are obtained based on the point cloud data; Determining a target local map based on the three-dimensional visual map and the three-dimensional laser map; determining a target map based on the target local map and a global map of the vehicle; storing a target map, determining a stable area and an unstable area in the target map, and storing a first validity period of the stable area and a second validity period of the unstable area in the target map; wherein the first validity period is greater than the second validity period; Wherein, determining a target local map based on the three-dimensional visual map and the three-dimensional laser map includes: Determining a first position of each target object in the three-dimensional visual map and a fusion weight of the first position; determining a second position of each target object in the three-dimensional laser map and a fusion weight of the second position; performing weighted fusion of the first position and the second position through a filter to obtain a target position of each target object; Based on each target object, the position of the target object in the target local map is located based on the target position of the target object, and rendering is performed at the target position based on the target feature of the target object to determine the target local map; wherein the target feature of the target object is obtained based on the weighted fusion of the image feature and the point cloud feature.

2. The method according to claim 1, characterized in that The determining of a target local map based on the three-dimensional visual map and the three-dimensional laser map includes: When the first condition is met, determining the three-dimensional visual map as the target local map; If the first condition is not met, determining the three-dimensional laser map as the target local map; The first condition is used to indicate that the light intensity in the vehicle driving environment meets a light intensity threshold.

3. The method according to claim 1 or 2, characterized in that The method further comprises: Recognizing the image data to obtain at least one static object and at least one dynamic object; The at least one identified dynamic object is filtered, and the at least one static object is determined as the target object.

4. The method according to claim 1 or 2, characterized in that In a case where the target objects include static objects and dynamic objects, constructing the three-dimensional visual map of the vehicle based on the target image features of the multiple target objects and the first positions of the multiple target objects includes: respectively determining a first influence coefficient of the static object and a second influence coefficient of the dynamic object; wherein the first influence coefficient is greater than the second influence coefficient; The three-dimensional visual map is constructed based on the target image features, the first position and the first influence coefficient of the static object, and the target image features, the first position and the second influence coefficient of the dynamic object.

5. The method according to claim 1 or 2, characterized in that Determining target image features of a plurality of target objects in the vehicle environment based on image data of the vehicle while traveling, including: Recognizing the image data of the vehicle in motion using a first network model to obtain initial feature maps of multiple target objects; Performing target-scale pooling processing on the initial feature maps of the multiple target objects through a second network model to obtain target-scale feature maps; The target scale feature map is concatenated with the initial feature map to obtain target image features of the multiple target objects.

6. The method according to claim 5, characterized in that In the case where the target scale includes multiple scales, the target image features include target image features at multiple scales; correspondingly, the three-dimensional visual map includes three-dimensional visual maps at multiple resolutions; The target map includes target maps of multiple resolutions; The method further comprises: When a second condition is met, assisting the vehicle in driving based on the target map of the first resolution; When a third condition is met, assisting the vehicle in driving based on the target map with a second resolution; The first resolution is lower than the second resolution, the second condition is used to characterize that the driving environment of the vehicle is an open environment; and the third condition is used to characterize that the driving environment of the vehicle is a narrow environment or a complex environment.

7. The method according to claim 1 or 2, characterized in that The determining, based on the image data and the point cloud data of the vehicle during driving, target image features of a plurality of target objects and target point cloud features of a plurality of target objects in the vehicle driving environment respectively includes: Acquiring initial image data and initial point cloud data of the vehicle while it is moving; Performing filtering processing on the initial image data and the initial point cloud data to obtain first image data and first point cloud data; performing abnormal data detection and correction on the first image data and the first point cloud data to obtain corrected second image data and second point cloud data; Temporally aligning the second image data and the second point cloud data using a time interpolation technique to obtain aligned third image data and third point cloud data; Based on the third image data and the third point cloud data of the vehicle during driving, target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle driving environment are respectively determined.

8. The method according to claim 1 or 2, characterized in that The method further comprises: Constructing a first node based on the vehicle posture and environmental features of the vehicle at each moment; the environmental features are obtained based on the image features and / or point cloud features; Determining a first constraint relationship between adjacent first nodes as a matching degree between environmental features of the adjacent first nodes; constructing a time graph based on the first nodes and the first constraint relationships between the first nodes; The target map is adjusted by adjusting the first nodes in the time graph and the constraint relationships between the first nodes.

9. The method according to claim 1 or 2, characterized in that The method further comprises: constructing a second node based on the different positions of the vehicle; Determine the edge between every two of the second nodes as a posture change value between every two of the second nodes; constructing a pose graph based on the second node and the edge between the second node; The second node and the edge between the second nodes in the pose graph are adjusted to adjust the target map.

10. The method according to claim 1 or 2, characterized in that The method further comprises: When it is detected that the second validity period of the unstable area has reached, the unstable area in the target map is reconstructed and updated.

11. The method according to claim 1 or 2, characterized in that The method further comprises: determining predicted behaviors of the plurality of target objects; If the predicted behavior of the target object affects the driving of the vehicle, a first prompt information is output on the target map; the first prompt information is used to prompt that the predicted behavior of the target object affects the driving of the vehicle.

12. A map determination device, characterized in that: The device comprises: a first determining unit, configured to respectively determine target image features of a plurality of target objects and target point cloud features of a plurality of target objects in a driving environment of the vehicle based on image data and point cloud data of the vehicle during driving; a first constructing unit, configured to construct a three-dimensional visual map of the vehicle based on target image features of the plurality of target objects and first positions of the plurality of target objects, wherein the first positions are obtained based on the image data; A second construction unit is configured to construct a three-dimensional laser map of the vehicle based on target point cloud features of the plurality of target objects and second positions of the plurality of target objects, wherein the second positions are obtained based on the point cloud data; a second determining unit, configured to determine a target local map based on the three-dimensional visual map and the three-dimensional laser map; and determine a target map based on the target local map and a global map of the vehicle; a storage unit configured to store a target map, determine a stable area and an unstable area in the target map, and store a first validity period of the stable area and a second validity period of the unstable area in the target map; the first validity period is greater than the second validity period; Wherein, determining a target local map based on the three-dimensional visual map and the three-dimensional laser map includes: Determining a first position of each target object in the three-dimensional visual map and a fusion weight of the first position; determining a second position of each target object in the three-dimensional laser map and a fusion weight of the second position; performing weighted fusion of the first position and the second position through a filter to obtain a target position of each target object; Based on each target object, the position of the target object in the target local map is located based on the target position of the target object, and rendering is performed at the target position based on the target feature of the target object to determine the target local map; wherein the target feature of the target object is obtained based on the weighted fusion of the image feature and the point cloud feature.

13. A vehicle, characterized in that: The vehicle includes a processor and a memory, wherein a computer program or instruction is stored in the memory, and when the computer program or instruction is executed by the processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that The storage medium stores a computer program or instructions, and when the computer program or instructions are executed by the processor, the method according to any one of claims 1 to 11 is implemented.

Images