Map determination method and device, vehicle and medium

Abstract

The invention relates to a map determination method and device, a vehicle and a medium, and the method at least comprises the steps: respectively determining the target image features of a plurality of target objects and the target point cloud features of the plurality of target objects in a vehicle driving environment based on the image data and point cloud data of the vehicle during driving; constructing a three-dimensional visual map of the vehicle based on the target image features of the plurality of target objects and the first positions of the plurality of target objects; constructing a three-dimensional laser map of the vehicle based on the target point cloud features of the plurality of target objects and the second positions of the plurality of target objects; determining a target map for assisting vehicle driving at least based on the three-dimensional visual map and the three-dimensional laser map; and storing the target map, determining a stable area and an unstable area in the target map, and storing the first validity period of the stable area and the second validity period of the unstable area in the target map. The target map determined by the scheme is more accurate.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a method, device, vehicle and medium for determining a map. Background Art

[0002] With the rapid development of autonomous driving and intelligent transportation technologies, how to obtain an accurate map has become one of the key factors for whether intelligent vehicles can operate normally in complex environments.

[0003] In the related art, visual maps are generally constructed directly based on the collected image data. In this way, in some complex scenes, such as rainy days or foggy scenes, the accuracy of the visual map is low, which may affect the driving of the vehicle and reduce the safety of the vehicle. Summary of the invention

[0004] One of the purposes of the present application is to provide a map determination method, device, vehicle and medium. In the scheme, a three-dimensional visual map and a three-dimensional laser map are constructed, and a target map is generated based on the three-dimensional visual map and the three-dimensional laser map, thereby improving the accuracy of the target map and the safety of driving, and improving the effectiveness of the target map display.

[0005] In order to achieve the above purpose, the technical solution adopted in this application is as follows: In a first aspect, the present application provides a method for determining a map, the method comprising: based on image data and point cloud data of a vehicle in motion, respectively determining target image features of multiple target objects and target point cloud features of multiple target objects in a vehicle driving environment; constructing a three-dimensional visual map of the vehicle based on the target image features of the multiple target objects and the first positions of the multiple target objects; the first position is obtained based on the image data; constructing a three-dimensional laser map of the vehicle based on the target point cloud features of the multiple target objects and the second positions of the multiple target objects; the second position is obtained based on the point cloud data; determining a target map for assisting vehicle driving based on at least the three-dimensional visual map and the three-dimensional laser map; storing the target map, determining stable areas in the target map, and storing a first validity period of the stable area and a second validity period of the unstable area in the target map; the first validity period is greater than the second validity period.

[0006] Based on the above technical means, a 3D visual map is first constructed based on image data, and a 3D laser map is constructed based on point cloud data, and then the target map is determined based on the 3D visual map and the 3D laser map. In this way, the target map can combine the advantages of the 3D visual map and the 3D laser map, thereby improving the accuracy and application scenarios of the target map and improving the safety of vehicle driving. In addition, the validity period of different areas can be clearly displayed, which improves the effectiveness of the target map display.

[0007] In one possible implementation, a target map for assisting vehicle driving is determined based at least on a three-dimensional visual map and a three-dimensional laser map, including: determining the target map based on the three-dimensional visual map and the three-dimensional laser map; or determining a target local map based on the three-dimensional visual map and the three-dimensional laser map; determining the target map based on the target local map and a global map of the vehicle.

[0008] Based on the above technical means, when determining the target map, if the target map is determined directly based on the three-dimensional visual map and the three-dimensional laser map, the advantages of both can be combined to improve the accuracy of the local map; when determining the target map, if it is determined based on the vehicle's global map, three-dimensional visual map and three-dimensional laser map, not only the accuracy of the local map is improved, but also the global planning is improved, the comprehensiveness of the target map is improved, and the accuracy of driving is improved.

[0009] In a possible implementation, a target local map is determined based on a three-dimensional visual map and a three-dimensional laser map, including: when a first condition is met, the three-dimensional visual map is determined as the target local map; when the first condition is not met, the three-dimensional laser map is determined as the target local map; wherein the first condition is used to characterize that the light intensity in the vehicle driving environment satisfies a light intensity threshold.

[0010] Based on the above technical means, when determining the target local map, it is determined according to the light intensity in the vehicle driving environment. When the light intensity meets the intensity threshold, the image information is more comprehensive and the obtained visual map is more accurate, so the target local map determined based on the three-dimensional visual map is also accurate. When the light intensity does not meet the intensity threshold, the image information is low, the obtained visual map loses more and the accuracy is low, so the target local map is determined based on the three-dimensional laser map. The target map obtained in this way can meet the needs of various scenarios and improve the accuracy and application scenarios.

[0011] In one possible implementation, a target local map is determined based on a three-dimensional visual map and a three-dimensional laser map, including: determining a first position of each target object in the three-dimensional visual map; determining a second position of each target object in the three-dimensional laser map; fusing the first position and the second position through a filter to obtain a target position of each target object; determining a target local map based on a target feature of each target object and the target position of each target object; wherein the target feature is obtained based on image features and / or point cloud features.

[0012] Based on the above technical means, when determining the local map of the target, the three-dimensional visual map and the three-dimensional laser map can also be fused. Since the map generally includes objects and their positions, the position of each object is fused to improve the accuracy of the position, and the features are fused to improve the accuracy of the features.

[0013] In a possible implementation, the method further includes: identifying the image data to obtain at least one static object and at least one dynamic object; filtering the identified at least one dynamic object and determining at least one static object as the target object.

[0014] Based on the above technical means, dynamic objects are filtered out. Since dynamic objects have dynamic errors, they will affect the accuracy of the map. The three-dimensional map determined based on static objects is more accurate.

[0015] In a possible implementation, when the target objects include static objects and dynamic objects, a three-dimensional visual map of the vehicle is constructed based on target image features of multiple target objects and first positions of multiple target objects, including: separately determining a first influence coefficient of the static object and a second influence coefficient of the dynamic object; the first influence coefficient is greater than the second influence coefficient; and constructing a three-dimensional visual map based on the target image features, first positions and first influence coefficients of the static objects, and the target image features, first positions and second influence coefficients of the dynamic objects.

[0016] Based on the above technical means, when the target object includes a static object and a dynamic object, by configuring the first influence coefficient and the second influence coefficient, the influence of the static object on the three-dimensional map is increased, and the influence of the dynamic object on the three-dimensional map is reduced, so that the obtained three-dimensional map is more accurate.

[0017] In a possible implementation, target image features of multiple target objects in a vehicle environment are determined based on image data of a moving vehicle, including: recognizing the image data of the moving vehicle through a first network model to obtain initial feature maps of the multiple target objects; performing target scale pooling processing on the initial feature maps of the multiple target objects through a second network model to obtain feature maps of the target scale; and concatenating the feature map of the target scale with the initial feature map to obtain target image features of the multiple target objects.

[0018] Based on the above technical means, when determining the target image features, the target scale can be configured based on actual needs, so that the target image features that meet the actual needs can be obtained.

[0019] In a possible implementation, when the target scale includes multiple scales, the target image features include target image features at multiple scales; correspondingly, the three-dimensional visual map includes three-dimensional visual maps of multiple resolutions; the target map includes target maps of multiple resolutions; the method also includes: when the second condition is met, assisting vehicle driving based on the target map of the first resolution; when the third condition is met, assisting vehicle driving based on the target map of the second resolution; wherein the first resolution is lower than the second resolution, the second condition is used to characterize that the vehicle's driving environment is an open environment; and the third condition is used to characterize that the vehicle's driving environment is a narrow environment or a complex environment.

[0020] Based on the above technical means, target maps with multiple resolutions can be obtained. Target maps with different resolutions can be called in different driving scenarios to meet the needs of various scenarios.

[0021] In a possible implementation, based on the image data and point cloud data of the vehicle while in motion, target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle's driving environment are determined respectively, including: acquiring initial image data and initial point cloud data of the vehicle while in motion; filtering the initial image data and initial point cloud data to obtain first image data and first point cloud data; detecting and correcting abnormal data in the first image data and the first point cloud data to obtain corrected second image data and second point cloud data; temporally aligning the second image data and the second point cloud data using a time interpolation technique to obtain aligned third image data and third point cloud data; and determining target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle's driving environment based on the third image data and the third point cloud data of the vehicle while in motion.

[0022] Based on the above technical means, the initial image data and initial point cloud data can be filtered, anomaly corrected, time aligned, etc., so that the processed data can better meet the needs, improve the accuracy of the data, thereby improving the accuracy of the target map and driving safety.

[0023] In a possible implementation, the method also includes: constructing a first node based on the vehicle posture and environmental features of the vehicle at each moment; the environmental features are obtained based on image features and / or point cloud features; determining the first constraint relationship between adjacent first nodes as the matching degree between environmental features of adjacent first nodes; constructing a time graph based on the first nodes and the first constraint relationship between the first nodes; adjusting the target map by adjusting the first nodes and the constraint relationship between the first nodes in the time graph.

[0024] Based on the above technical means, the target map can be adjusted. The adjustment based on the time map can reduce the time accumulation error, thereby improving the accuracy of the target map and the safety of driving.

[0025] In a possible implementation, the method further includes: constructing a second node based on different positions of the vehicle; determining the edge between each two second nodes as the posture change value between each two second nodes; constructing a pose graph based on the edge between the second node and the second node; adjusting the second node and the edge between the second nodes in the pose graph, and adjusting the target map.

[0026] Based on the above technical means, the target map can be adjusted. Adjustment based on the pose graph can reduce the cumulative error caused by position changes, thereby improving the accuracy of the target map and driving safety.

[0027] In a possible implementation, the method further includes: when it is detected that the second validity period of the unstable area has reached, reconstructing and updating the unstable area in the target map.

[0028] Based on the above technical means, the map information of the unstable area can be updated in time, achieving high precision of the target map while also having high real-time performance.

[0029] In a possible implementation, the method further includes: determining predicted behaviors of multiple target objects; if the predicted behaviors of the target objects affect the driving of the vehicle, outputting first prompt information on the target map; the first prompt information is used to prompt that the predicted behaviors of the target objects affect the driving of the vehicle.

[0030] Based on the above technical means, behaviors that affect vehicle driving can be output in the target map, which improves the richness of the target map display and the safety of vehicle driving.

[0031] In a second aspect, the present application provides a map determination device, the device comprising: A first determination unit is used to determine target image features of multiple target objects and target point cloud features of multiple target objects in a driving environment of the vehicle, based on image data and point cloud data of the vehicle during driving; A first construction unit is used to construct a three-dimensional visual map of the vehicle based on target image features of the plurality of target objects and first positions of the plurality of target objects; the first positions are obtained based on the image data; A second construction unit is used to construct a three-dimensional laser map of the vehicle based on target point cloud features of the multiple target objects and second positions of the multiple target objects; the second position is obtained based on the point cloud data; A second determination unit, configured to determine a target map for assisting vehicle driving based at least on the three-dimensional visual map and the three-dimensional laser map; The storage unit is used to store the target map, determine the stable area and the unstable area in the target map, and store the first validity period of the stable area and the second validity period of the unstable area in the target map; the first validity period is greater than the second validity period.

[0032] In a third aspect, the present application further provides a vehicle, comprising a processor and a memory, wherein the memory stores a computer program or instructions, and the computer program or instructions, when executed by the processor, implement the method provided in the first aspect.

[0033] In a fourth aspect, the present application further provides a storage medium having a computer program or instruction stored thereon, and the computer program or instruction, when executed by a processor, implements the method provided in the first aspect above.

[0034] In a fifth aspect, the present application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the method provided in the first aspect is implemented.

[0035] It should be noted that the technical effects of the second to fifth aspects can refer to the detailed description of the first aspect above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a first optional flow chart of a method for determining a map provided in an embodiment of the present application; Figure 2 A second optional flow chart of the method for determining a map provided in an embodiment of the present application; Figure 3 A third optional flow chart of the method for determining a map provided in an embodiment of the present application; Figure 4 A fourth optional flow chart of the method for determining a map provided in an embodiment of the present application; Figure 5 A fifth optional flow chart of the method for determining a map provided in the embodiment of the present application; Figure 6 A sixth optional flow chart of the method for determining a map provided in an embodiment of the present application; Figure 7 A seventh optional flow chart of the method for determining a map provided in the embodiment of the present application; Figure 8 An eighth optional flow chart of the method for determining a map provided in an embodiment of the present application; Fig. 9 A ninth optional flow chart of the method for determining a map provided in an embodiment of the present application; Fig.10 A tenth optional flow chart of the method for determining a map provided in an embodiment of the present application; Fig.11 A schematic diagram of an eleventh optional flow chart of the method for determining a map provided in an embodiment of the present application; Fig.12 An optional structural diagram of a data processing related module provided in an embodiment of the present application; Fig.13 An optional flowchart of a positioning process provided in an embodiment of the present application; Fig.14 An optional flowchart of a scene understanding process provided in an embodiment of the present application; Fig.15 A schematic diagram of an optional process of positioning and map construction provided in an embodiment of the present application; Fig.16 An optional flowchart of a map optimization process provided in an embodiment of the present application; Fig.17 An optional flowchart of a map storage process provided in an embodiment of the present application; Fig.18 An optional structural diagram of a map determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first\second\third" are used only as examples to distinguish different objects, and do not represent a specific order for the objects, nor do they have a limitation on the order of precedence. It is understandable that "first\second\third" can be interchanged with a specific order or order of precedence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0041] The embodiments of the present application provide a method, device, vehicle, medium, and product for determining a map. The method for determining a map is performed by a map determining device, and the map determining device can be deployed on a vehicle. The following describes various embodiments of the method, device, vehicle, medium, and product for determining a map provided in the embodiments of the present application.

[0042] In a first aspect, an embodiment of the present application provides a method for determining a map.

[0043] refer to Figure 1 The process may include but is not limited to S101 to S105. The process may be implemented by a local device on the vehicle side, or by a cloud connected to the vehicle side, or by a combination of the vehicle side and the cloud side, and by interaction between the vehicle side and the cloud side.

[0044] S101 . Based on image data and point cloud data of a vehicle in motion, respectively determine target image features of a plurality of target objects and target point cloud features of a plurality of target objects in a vehicle driving environment.

[0045] The embodiment of the present application does not limit the type of vehicle and can be configured according to actual needs. For example, the vehicle may include but is not limited to: a gasoline vehicle, an electric vehicle, etc. The vehicle here may be a vehicle with an assisted driving function or a vehicle without an assisted driving function.

[0046] Image data refers to data collected by a camera. Point cloud data refers to data collected by a laser radar or millimeter wave radar. The image data and point cloud data here can be the collected raw data or the data after processing (filtering, etc.) the raw data.

[0047] The vehicle driving environment refers to the external environment during the vehicle driving process.

[0048] The target object refers to an object that meets the requirements in the vehicle driving environment. For example, the target object can be all objects or only static objects.

[0049] Image features are used to characterize object information in an image, and target image features are used to characterize target object information in an image.

[0050] Point cloud features are used to characterize object information in the point cloud, and target image features are used to characterize target object information in the point cloud.

[0051] S101 can be implemented as follows: after acquiring image data of a moving vehicle, the image data is processed by a target recognition algorithm to identify multiple target objects, and then feature extraction is performed on the image data corresponding to the multiple target objects by a feature extraction algorithm to obtain target image features of the multiple target objects; after acquiring point cloud data of a moving vehicle, the point cloud data is processed by a target recognition algorithm to identify multiple target objects, and then feature extraction is performed on the point cloud data of the multiple target objects by a feature extraction algorithm to obtain target point cloud features of the multiple target objects.

[0052] The recognition algorithm in the image data processing process may be the same as or different from the recognition algorithm in the point cloud data processing process. The feature extraction algorithm in the image data processing process may be the same as or different from the feature extraction algorithm in the point cloud data processing process.

[0053] S102: construct a three-dimensional visual map of the vehicle based on target image features of the multiple target objects and the first positions of the multiple target objects.

[0054] The first position is obtained based on the image data. The embodiment of the present application does not limit the manner in which the first position is determined based on the image data, and can be configured according to actual needs. For example, the first position can be determined based on image data at two different angles.

[0055] The target objects here include objects in the vehicle's driving environment, such as pedestrians, buildings, etc., and also the vehicle itself.

[0056] S102 can be implemented as follows: construct a three-dimensional visual map based on the first position of the vehicle, first obtain the first position of the vehicle and the target image features of the vehicle, and render the vehicle in the three-dimensional visual map at the first position of the vehicle based on the target image features of the vehicle; then obtain the first positions of other target objects and the target image features of other target objects, and render each other target object in the three-dimensional visual map at the first position of each other target object based on the image features of each other target object, thereby obtaining a three-dimensional visual map of the vehicle.

[0057] S103: construct a three-dimensional laser map of the vehicle based on the target point cloud features of the multiple target objects and the second positions of the multiple target objects.

[0058] The second position is obtained based on the point cloud data. The embodiment of the present application does not limit the method of determining the second position based on the image data, and can be configured according to actual needs. For example, the second position can be determined based on point cloud data at two different times.

[0059] S103 can be implemented as follows: construct a three-dimensional laser map based on the second position of the vehicle, first obtain the second position of the vehicle and the target point cloud features of the vehicle, and render the vehicle in the three-dimensional laser map at the second position of the vehicle based on the target point cloud features of the vehicle; then obtain the second positions of other target objects and the target point cloud features of other target objects, and render each other target object in the three-dimensional laser map at the second position of each other target object based on the point cloud features of each other target object, thereby obtaining a three-dimensional laser map of the vehicle.

[0060] S104. Determine a target map for assisting vehicle driving based at least on the three-dimensional visual map and the three-dimensional laser map.

[0061] In a possible implementation, a target map for assisting vehicle driving is determined based on the three-dimensional visual map and the three-dimensional laser map, so that the target map obtained is more accurate.

[0062] In another possible implementation, S104 may be implemented as follows: based on the global map, the 3D visual map and the 3D laser map, a target map for assisting vehicle driving is determined. For example, a target local map is first determined based on the 3D visual map and the 3D laser map, and then the target local map is spliced ​​and fused with the global map to obtain the target map. The target map obtained in this way is not only accurate but also comprehensive.

[0063] In another possible implementation, S104 may be implemented as follows: a target map for assisting vehicle driving may also be determined based on inertial data, a global map, a 3D visual map, and a 3D laser map. For example, a target local map is first determined based on the 3D visual map and the 3D laser map, and then the target local map is spliced ​​and fused with the global map to obtain a fused map; and then the fused map is optimized based on the inertial data. The target map obtained in this way can be applied in a weak network environment. For example, in a tunnel scenario, the position of the vehicle can be determined based on the inertial data, thereby updating the target map.

[0064] S105 , storing the target map, determining the stable area in the target map, and storing the first validity period of the stable area and the second validity period of the unstable area in the target map.

[0065] The first validity period is greater than the second validity period.

[0066] The embodiment of the present application does not limit the location of storing the target map, and can be configured according to actual needs. For example, it can be stored in the local end of the vehicle, or in the cloud, or of course, it can be stored in both ends at the same time.

[0067] Here, the target map can be stored in a designated location on the local side of the vehicle, or in a designated location in the cloud.

[0068] Stable areas are areas whose location and appearance do not change over a long period of time, such as residential areas.

[0069] Unstable areas are areas whose location and appearance may change over time, such as construction areas.

[0070] The embodiment of the present application does not limit the duration of the first validity period and the second validity period, and can be configured according to actual needs. For example, the second validity period can be configured according to the construction period.

[0071] S105 can be implemented as follows: storing the target map at a designated location, and then determining the stable area and the unstable area in the target map based on the user's input operation. For the stable area in the target map, a first validity period of the stable area is added based on the user's operation; for the unstable area, a second validity period of the unstable area is added based on the user's operation.

[0072] In this embodiment, the method includes: based on image data and point cloud data of the vehicle during driving, respectively determining target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle driving environment; based on the target image features of the multiple target objects and the first positions of the multiple target objects, constructing a three-dimensional visual map of the vehicle; the first position is obtained based on the image data; based on the target point cloud features of the multiple target objects and the second positions of the multiple target objects, constructing a three-dimensional laser map of the vehicle; the second position is obtained based on the point cloud data; and determining a target map for assisting vehicle driving based at least on the three-dimensional visual map and the three-dimensional laser map.

[0073] Based on the above technical means, a 3D visual map is first constructed based on image data, and a 3D laser map is constructed based on point cloud data, and then the target map is determined based on the 3D visual map and the 3D laser map. In this way, the target map can combine the advantages of the 3D visual map and the 3D laser map, thereby improving the accuracy and application scenarios of the target map and improving the safety of vehicle driving. In addition, the validity period of different areas can be clearly displayed, which improves the effectiveness of the target map display and improves the user experience.

[0074] Next, the process of determining the target map for assisting vehicle driving based on at least the three-dimensional visual map and the three-dimensional laser map in S104 is described.

[0075] The process may include but is not limited to implementation 1 or implementation 2 described below.

[0076] 1. Determine the target map based on the 3D visual map and the 3D laser map.

[0077] For example, one of the three-dimensional visual map and the three-dimensional laser map can be selected as the target map based on different scenarios. For another example, the three-dimensional visual map and the three-dimensional laser map can be fused to obtain the target map.

[0078] Implementation 2: Determine the target local map based on the 3D visual map and the 3D laser map; determine the target map based on the target local map and the global map of the vehicle.

[0079] Since image data and point cloud data are collected on the spot and the detection distance of cameras and radar sensors is limited, the obtained three-dimensional visual map and three-dimensional laser map are both local maps. The global map can be obtained based on navigation data.

[0080] The target local map can be determined based on the three-dimensional visual map and the three-dimensional laser map, and then the corresponding area in the entire map can be updated based on the target local map to obtain the target map.

[0081] Based on the above technical means, when determining the target map, if the target map is determined directly based on the three-dimensional visual map and the three-dimensional laser map, the advantages of both can be combined to improve the accuracy of the local map; when determining the target map, if it is determined based on the vehicle's global map, three-dimensional visual map and three-dimensional laser map, not only the accuracy of the local map is improved, but also the global planning is improved, the comprehensiveness of the target map is improved, and the accuracy of driving is improved.

[0082] Next, the process of determining the target local map based on the three-dimensional visual map and the three-dimensional laser map in implementation 2 is described.

[0083] The process may include but is not limited to the following method 1 or method 2.

[0084] Method 1: Determine the target local map based on the switching mechanism; Method 2: Determine the target local map based on the fusion mechanism.

[0085] Next, the process of determining the target local map based on the switching mechanism in method 1 is described.

[0086] When the first condition is met, the three-dimensional visual map is determined as the target local map; when the first condition is not met, the three-dimensional laser map is determined as the target local map; wherein the first condition is used to characterize that the light intensity in the vehicle driving environment meets the light intensity threshold.

[0087] In a driving environment that meets the light intensity threshold, the image data collected by the camera is clearer and the resulting three-dimensional visual map is more accurate.

[0088] The embodiment of the present application does not limit the value of the light intensity threshold and can be configured according to actual needs.

[0089] Based on the above technical means, when determining the target local map, it is determined according to the light intensity in the vehicle driving environment. When the light intensity meets the intensity threshold, the image information is more comprehensive and the obtained visual map is more accurate, so the target local map determined based on the three-dimensional visual map is also accurate. When the light intensity does not meet the intensity threshold, the image information is low, the obtained visual map loses more and the accuracy is low, so the target local map is determined based on the three-dimensional laser map. The target map obtained in this way can meet the needs of various scenarios and improve the accuracy and application scenarios.

[0090] Next, the process of determining the target local map based on the fusion mechanism in method 2 is described.

[0091] refer to Figure 2 As shown in the content, the process may include but is not limited to the following S201 to S204.

[0092] S201. Determine a first position of each target object in the three-dimensional visual map.

[0093] S201 can be implemented as follows: determining the position of each target object based on the image data to obtain the first position of each target object. The method for determining the position based on the image in the embodiment of the present application is not limited and can be configured according to actual needs. For example, the first position of the target object can be determined based on images of the target object at different angles.

[0094] S202: Determine a second position of each target object in the three-dimensional laser map.

[0095] S202 may be implemented as follows: determining the position of each target object based on the point cloud data to obtain the second position of each target object. Since the point cloud information is distance information, the second position of the target object may be obtained by performing a reference position transformation on the point cloud data.

[0096] S203: Fusing the first position and the second position through a filter to obtain a target position of each target object.

[0097] The embodiment of the present application does not limit the type of filter and can be configured according to actual needs.

[0098] For example, the filter may be an extended Kalman filter.

[0099] S203 may be implemented as follows: inputting the first position and the second position into a filter, fusing the first position and the second position through processing by the filter, and obtaining the target position of each target object. Here, a fusion weight of the first position and the second position may also be configured in the filter.

[0100] S204: Determine a target local map based on the target feature of each target object and the target position of each target object.

[0101] The target features are obtained based on image features and / or point cloud features.

[0102] The target feature can be an image feature, a point cloud feature, or a feature that is a fusion of the image feature and the point cloud feature.

[0103] The fusion of image features and point cloud features can be configured with different weights for image features and point cloud features, and then feature fusion can be performed. The weight here can be shared by all target objects, or different weights can be configured for different objects. For example, if an object is covered by a shadow and the image feature data is less, the weight of the point cloud feature can be configured to be slightly larger.

[0104] S204 can be implemented as follows: for each target object, the position of the target object in the target local map is located by the target position of the target object, and then at the target position, the target object is rendered based on the target features of the target object, thereby forming a target local map.

[0105] Here you can configure the granularity of target object rendering according to actual needs.

[0106] Based on the above technical means, when determining the local map of the target, the three-dimensional visual map and the three-dimensional laser map can also be fused. Since the map generally includes objects and their positions, the position of each object is fused to improve the accuracy of the position, and the features are fused to improve the accuracy of the features.

[0107] It should be noted that the target map in implementation 1 may be determined by referring to the method of determining the target local map, which will not be described in detail here.

[0108] Next, the process of obtaining the target object is described.

[0109] In one possible implementation, reference Figure 3 As shown in the content, the process may include but is not limited to the following S301 and S302.

[0110] S301: Identify image data to obtain at least one static object and at least one dynamic object.

[0111] S301 may be implemented as follows: using an image recognition algorithm to process the image data, identifying objects in the image data, and then using a classification model to determine the type of each object, thereby obtaining at least one static object and at least one dynamic object.

[0112] S302: Filter at least one identified dynamic object and determine at least one static object as a target object.

[0113] That is, the position and feature information of dynamic objects are filtered out.

[0114] S302 may be implemented as follows: filtering the at least one identified dynamic object, filtering out the position information and feature information of the dynamic object, and determining the at least one static object after filtering as the target object.

[0115] Based on the above technical means, dynamic objects are filtered out. Since dynamic objects have dynamic errors, they will affect the accuracy of the map. The three-dimensional map determined based on static objects is more accurate.

[0116] In another possible implementation, both static objects and dynamic objects may be determined as target objects.

[0117] Next, the implementation process of constructing the three-dimensional visual map of the vehicle based on the target image features of the multiple target objects and the first positions of the multiple target objects in 102 is described.

[0118] In a possible implementation manner, when the target object includes a static object and a dynamic object, reference Figure 4 As shown in the content, the process may include but is not limited to the following S401 and S402.

[0119] S401 : Determine a first influence coefficient of a static object and a second influence coefficient of a dynamic object respectively.

[0120] The first influence coefficient is greater than the second influence coefficient.

[0121] The embodiment of the present application does not limit the values ​​of the first influence coefficient and the second influence coefficient, and can be configured according to actual needs.

[0122] S401 may be implemented as follows: determining the first influence coefficient of the static object and the second influence coefficient of the dynamic object respectively according to the user input; or determining the first influence coefficient of the static object and the second influence coefficient of the dynamic object respectively based on the influence of the static object and the dynamic object on the result.

[0123] S402: Construct a three-dimensional visual map based on the target image features, the first position and the first influence coefficient of the static object and the target image features, the first position and the second influence coefficient of the dynamic object.

[0124] S402 can be implemented as follows: for each static object, the display of the static object in the three-dimensional visual map is determined based on the first influence coefficient, the first position and the target image feature; for each dynamic object, the display of the dynamic object in the three-dimensional visual map is determined based on the second influence coefficient, the first position and the target image feature.

[0125] Based on the above technical means, when the target object includes a static object and a dynamic object, by configuring the first influence coefficient and the second influence coefficient, the influence of the static object on the three-dimensional map is increased, and the influence of the dynamic object on the three-dimensional map is reduced, so that the obtained three-dimensional map is more accurate.

[0126] In another possible implementation, when the target objects include only static objects, the three-dimensional visual map is determined based only on dynamic objects.

[0127] Next, the process of respectively determining target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle driving environment based on the image data and point cloud data of the vehicle during driving in S101 is described.

[0128] The process of determining image features and point cloud features is similar, and the following is an example of determining image features. The process of determining point cloud features can refer to the description of the image feature process.

[0129] refer to Figure 5 As shown in the content, the process may include but is not limited to the following S501 to S503.

[0130] S501. Recognize image data of a moving vehicle using a first network model to obtain initial feature maps of multiple target objects.

[0131] The embodiment of the present application does not limit the type of the first network model, and can be configured according to actual needs. For example, the first network model can be a multi-target detection model.

[0132] S501 can be implemented as follows: inputting an image of a moving vehicle into a first network model, performing recognition processing on the image data of the moving vehicle through the first network model, and obtaining initial feature maps of multiple target objects.

[0133] S502: Perform target-scale pooling processing on initial feature maps of multiple target objects through a second network model to obtain target-scale feature maps.

[0134] For example, the second network model may be a DeepLab model version 3 (DeepLabv3). For example, the second network model may include: a model consisting of a backbone network and a spatial pyramid pooling module.

[0135] The embodiment of the present application does not limit the size of the target scale, and can be configured according to actual needs. The target scale here can be one scale or multiple scales.

[0136] S502 may be implemented as follows: inputting the initial feature maps of the plurality of target objects into the second network model, performing target scale pooling processing on the initial feature maps of the plurality of target objects through the second network model, and obtaining target scale feature maps. The target scale is pre-configured in the second network model.

[0137] S503: Concatenate the feature map of the target scale with the initial feature map to obtain target image features of multiple target objects.

[0138] S503 may be implemented as follows: performing feature map splicing processing on the feature map of the target scale and the initial feature map to obtain target image features of multiple target objects. If the target scale is multiple scales, the feature maps of the multiple scales are spliced ​​with the initial feature map respectively.

[0139] Based on the above technical means, when determining the target image features, the target scale can be configured based on actual needs, so that the target image features that meet the actual needs can be obtained.

[0140] Among them, the target scale can be configured according to actual needs. When the target scale includes multiple scales, the target image features include target image features at multiple scales; correspondingly, the three-dimensional visual map includes three-dimensional visual maps of multiple resolutions; and the target map includes target maps of multiple resolutions.

[0141] refer to Figure 6 The content shown, the map determination method provided by this embodiment may also include but is not limited to the following S601 and S602.

[0142] S601. When a second condition is met, assist vehicle driving based on a target map with a first resolution.

[0143] The second condition is used to characterize that the driving environment of the vehicle is an open environment. The embodiment of the present application does not limit the method of determining an open environment, and can be configured according to actual needs. Among them, it can be determined based on image data; for example, when grasslands, plains and other scenes are detected, it is determined to be an open environment. It can also be determined based on point cloud data; for example, when the average distance of the detected objects is greater than the distance threshold, it is determined that the current driving is in an open environment.

[0144] The embodiment of the present application does not limit the value of the first resolution and can be configured according to actual needs.

[0145] S601 may be implemented as: when the second condition is met, calling the target map of the first resolution, and outputting the target map of the first resolution to assist the vehicle in driving. In this way, system resources can be saved.

[0146] S602: When the third condition is met, assist vehicle driving based on the target map with the second resolution.

[0147] The first resolution is lower than the second resolution.

[0148] The third condition is used to characterize that the driving environment of the vehicle is a narrow environment or a complex environment.

[0149] The embodiment of the present application does not limit the method of determining an open environment, and can be configured according to actual needs. Among them, it can be determined based on image data; for example, when various high-rise buildings, pedestrians, traffic lights and other scenes are detected, it is determined to be a narrow environment or a complex environment. It can also be determined based on point cloud data; for example, when the average distance of the detected objects is less than the distance threshold, it is determined that the current driving is in a narrow environment or a complex environment.

[0150] The embodiment of the present application does not limit the value of the second resolution and can be configured according to actual needs.

[0151] In simple terms, the second resolution may also be referred to as high resolution, and the first resolution may also be referred to as low resolution.

[0152] S602 may be implemented as: when the third condition is met, calling the target map of the second resolution, and outputting the target map of the second resolution to assist the vehicle in driving. In this way, the needs of actual scenarios can be met.

[0153] Based on the above technical means, target maps with multiple resolutions can be obtained. Target maps with different resolutions can be called in different driving scenarios to meet the needs of various scenarios.

[0154] The map determination method provided in the embodiment of the present application can also process image data and point cloud data, and determine image features and point cloud features based on the processed image data and point cloud data.

[0155] refer to Figure 7 As shown in the content, the process may include but is not limited to the following S701 to S705.

[0156] S701, obtaining initial image data and initial point cloud data of a vehicle in motion.

[0157] The initial image data here is the image data collected by the camera. The embodiment of the present application does not limit the camera that collects the initial image data, and can be configured according to actual needs. For example, it can be the camera in front of the car. Or, it can also be all cameras on the car for collecting external images.

[0158] The initial point cloud data here is the point cloud data collected by the radar. The embodiment of the present application does not limit the radar for collecting the initial point cloud data, and can be configured according to actual needs. For example, it can be a millimeter wave radar and / or a laser radar. The deployment location of the radar can also be configured and selected according to actual needs.

[0159] S701 can be implemented as follows: collecting initial image data and initial point cloud data of the vehicle while driving; or storing the initial image data and initial point cloud data of the vehicle while driving after collecting, and directly calling the stored initial image data and initial point cloud data here.

[0160] S702: Filter the initial image data and the initial point cloud data to obtain first image data and first point cloud data.

[0161] The embodiment of the present application does not limit the filtering algorithm and can be configured according to actual needs.

[0162] The filtering method may include but is not limited to one or more of the following: low-pass filtering, Kalman filtering, mean filtering, median filtering, extended Kalman filtering, and unscented Kalman filtering.

[0163] S702 may be implemented as follows: the electronic device performs filtering processing on the initial image data and the initial point cloud data respectively through one or more of the above filtering methods to obtain first image data and first point cloud data.

[0164] S703 , detecting and correcting abnormal data on the first image data and the first point cloud data to obtain corrected second image data and second point cloud data.

[0165] The embodiments of the present application do not limit the detection and correction algorithm of abnormal data and can be configured according to actual needs.

[0166] The detection of abnormal data can be achieved through threshold detection and time series analysis. For example, by setting a reasonable threshold range, when the sensor data exceeds the normal range, the system will mark it as abnormal data and trigger the correction process; using time series analysis methods including algorithms such as moving average and autoregressive models, by analyzing the historical change trend of the data and identifying abnormal points, it can effectively detect short-term mutations and long-term drift data, thereby triggering the corresponding correction strategy.

[0167] The detection of abnormal data can also be achieved through statistical anomaly detection algorithms. For example, based on the 3σ rule of mean and variance, the mean and standard deviation of the collected sensor data are calculated. If a data point deviates from the mean by more than 3 times the standard deviation, the data is considered abnormal, which improves the robustness of the system.

[0168] Data correction can be achieved by resampling abnormal data points and obtaining new sensor data to replace the abnormal points.

[0169] For data correction, linear interpolation or spline interpolation can also be used to generate new data through surrounding data points and replace outliers.

[0170] Data correction can also be achieved by using historical data for retrospective analysis and correcting current data through weighted average or prediction models.

[0171] S704: Time-align the second image data and the second point cloud data using a time interpolation technique to obtain aligned third image data and third point cloud data.

[0172] The embodiment of the present application does not specifically limit the method of time alignment and can be configured according to actual needs.

[0173] For example, the data collected by each sensor is accompanied by timestamp information. The data is synchronized through the timestamp and data with different sampling rates are converted to the same time axis. Before multi-sensor data fusion, the system aligns the data. The data alignment uses timestamp interpolation technology to align the output of each sensor in a way that minimizes time delay.

[0174] In order to avoid state lag caused by data delay, a delay compensation mechanism can also be introduced to estimate data delay through a prediction model and make state predictions based on the estimated value. For example, when image data lags behind point cloud data, a short-term prediction can be made based on the change trend of image data to compensate for the impact of its delay; delay optimization needs to be adaptively updated according to the driving environment and sensor characteristics. The system will regularly detect changes in data delay and dynamically adjust delay compensation parameters to ensure the real-time performance of the system in different environments.

[0175] S705 . Based on the third image data and the third point cloud data of the driving vehicle, respectively determine target image features of multiple target objects and target point cloud features of multiple target objects in the driving environment of the vehicle.

[0176] The implementation of S705 is similar to that of S101, and will not be described in detail here.

[0177] Through the above data optimization and processing flow, the accuracy and real-time performance of multi-sensor fusion data can be ensured, providing data support for building accurate maps.

[0178] Based on the above technical means, the initial image data and initial point cloud data can be filtered, anomaly corrected, time aligned, etc., so that the processed data can better meet the needs, improve the accuracy of the data, thereby improving the accuracy of the target map and driving safety.

[0179] The map determination method provided in the embodiment of the present application also includes a target map adjustment process.

[0180] The adjustment process may include but is not limited to the following adjustment process based on a time graph and an adjustment process based on a pose graph.

[0181] refer to Figure 8 As shown in the content, the adjustment process based on the time diagram may include but is not limited to the following S801 to S804.

[0182] S801: Construct a first node based on the vehicle posture and environmental characteristics of the vehicle at each moment.

[0183] The environmental features are obtained based on image features and / or point cloud features.

[0184] The environmental features may be image features and / or point cloud features, or may be features fused together based on their respective weights.

[0185] S801 may be implemented as follows: based on a time graph construction algorithm, a first node is constructed for the vehicle posture and environmental features of the vehicle at each moment.

[0186] S802: Determine that a first constraint relationship between adjacent first nodes is a matching degree between environmental features of adjacent first nodes.

[0187] The matching degree between the environmental features of each two first nodes is determined, so as to obtain the first constraint relationship between each two first nodes. Here, each two first nodes are first nodes corresponding to the connected moment.

[0188] S802 can be implemented as follows: based on the time graph construction algorithm, for adjacent first nodes, first determine the matching degree between the environmental features of the adjacent first nodes, and then add the first constraint relationship between the adjacent first nodes based on the matching degree between the environmental features of the adjacent first nodes.

[0189] S803: Construct a time graph based on the first nodes and the first constraint relationships between the first nodes.

[0190] A time graph is constructed based on the first nodes constructed at all moments and the first constraint associations between the first nodes corresponding to every two adjacent moments.

[0191] S803 can be implemented as follows: the electronic device traverses the vehicle posture and environmental characteristics of the vehicle at each moment, constructs all first nodes, traverses every two first nodes, and adds a first constraint relationship according to the matching degree of the environmental characteristics between every two first nodes, thereby obtaining a time graph.

[0192] S804: Adjust the target map by adjusting the first nodes in the time graph and the constraint relationship between the first nodes.

[0193] The positioning error is reduced by optimizing the positions of nodes and the constraints between edges.

[0194] S804 may be implemented as follows: by adjusting the first nodes in the time graph and the constraint relationship between the first nodes, the positioning error is reduced to optimize the target map.

[0195] Based on the above technical means, the target map can be adjusted. The adjustment based on the time map can reduce the time accumulation error, thereby improving the accuracy of the target map and the safety of driving.

[0196] refer to Fig. 9 As shown in the content, the adjustment process based on the pose graph may include but is not limited to the following S901 to S904.

[0197] S901. Construct a second node based on different positions of the vehicle.

[0198] The locations where the vehicles travel are determined to construct a plurality of second nodes.

[0199] S901 may be implemented as follows: based on a pose graph construction algorithm, a second node is constructed for each driving position of the vehicle, and different second nodes are constructed for different positions of the vehicle.

[0200] For example, the relative pose between the loop node and the history node is added to the pose graph as a new constraint, a new loop constraint is generated, and the position is marked.

[0201] S902. Determine the edge between every two second nodes as the posture change value between every two second nodes.

[0202] Every two second nodes correspond to two second nodes corresponding to two positions whose position change process is always continuous.

[0203] An edge between every two second nodes is determined based on a pose change value between every two second nodes.

[0204] S902 can be implemented as follows: based on the pose graph construction algorithm, for adjacent second nodes, first determine the pose change value between every two second nodes, and then add the pose change value between every two second nodes to the edge between adjacent second nodes.

[0205] The confirmed loop constraints can also be added to the pose graph to strengthen the relative position constraints between keyframes (a map of moments), adding a node for each keyframe and adding constraint edges between adjacent keyframes to reduce the cumulative error.

[0206] S903: Construct a pose graph based on the second node and the edge between the second node.

[0207] A pose graph is constructed based on all second nodes and the edges between every two second nodes.

[0208] S903 can be implemented as follows: the electronic device traverses each driving position of the vehicle to construct all second nodes, traverses the position change values ​​between all second nodes to construct edges between the second nodes and the second nodes, and obtains a posture graph.

[0209] S904: Adjust the second node and the edge between the second nodes in the pose graph, and adjust the target map.

[0210] S904 may be implemented as follows: using a graph optimization algorithm to adjust the second node and the edges between the second nodes in the pose graph by minimizing the error between nodes, thereby achieving target map optimization.

[0211] Here, if the vehicle passes through the historical area, a new constraint is added after loop confirmation.

[0212] Based on the above technical means, the target map can be adjusted. Adjustment based on the pose graph can reduce the cumulative error caused by position changes, thereby improving the accuracy of the target map and driving safety.

[0213] The map determination method provided in the embodiment of the present application may also include updating the map.

[0214] refer to Fig.10 The update process may include but is not limited to the following S106.

[0215] S106: When it is detected that the second validity period of the unstable area has arrived, reconstruct and update the unstable area in the target map.

[0216] The embodiment of the present application does not limit the method of reconstructing and updating the map of the unstable area, and reference may be made to the description of the above creation process.

[0217] Based on the above technical means, the map information of the unstable area can be updated in time, achieving high precision of the target map while also having high real-time performance.

[0218] The map determination method provided in the embodiment of the present application may also include a prompting process for the predicted behavior of the target object.

[0219] refer to Fig.11 As shown in the content, the process may include but is not limited to the following S1101 and S1102.

[0220] S1101. Determine predicted behaviors of multiple target objects.

[0221] The predicted behavior of the target object can also be predicted through behavior prediction algorithms.

[0222] For example, in a complex traffic environment, there may be multiple dynamic traffic participants (such as pedestrians, bicycles, other vehicles, etc.). The behavior prediction model can be based on multi-target tracking technology to track the trajectories of different target objects and combine the semantic information of the environment to predict the future position of each target, thereby determining whether the trajectory of the target object affects the driving of the vehicle, thereby improving the overall prediction ability of the system.

[0223] S1101 may be implemented as follows: predicting behaviors of multiple target objects by using a prediction model or a prediction algorithm, thereby obtaining predicted behaviors of the multiple target objects.

[0224] S1102: If the predicted behavior of the target object affects the driving of the vehicle, output a first prompt message on the target map.

[0225] S1102 may be implemented as follows: matching the trajectory of the predicted behavior of each target object with the predicted trajectory of the vehicle to determine whether the predicted behavior of the target object affects the driving of the vehicle; if it is determined that the predicted behavior of the target object affects the driving of the vehicle, outputting first prompt information on the target map. If it is determined that the predicted behavior of the target object does not affect the driving of the vehicle, no prompt is required.

[0226] The first prompt information is used to prompt that the predicted behavior of the target object affects the driving of the vehicle.

[0227] The embodiment of the present application does not limit the prompting method of the first prompting information, and can be configured according to actual needs. For example, the first prompting information can be a voice prompt. Alternatively, the target object that affects the driving of the vehicle can also be highlighted.

[0228] Based on the above technical means, behaviors that affect vehicle driving can be output in the target map, which improves the richness of the target map display and the safety of vehicle driving.

[0229] It should be noted that the descriptions of the above embodiments and steps can be combined according to actual needs if there is no contradiction, and will not be repeated here one by one.

[0230] Below, taking an assisted driving vehicle as an example, the process of determining the map provided in this embodiment of the present application is explained.

[0231] With the rapid development of autonomous driving and intelligent transportation technologies, accurate positioning has become one of the key factors for the normal operation of intelligent vehicles in complex environments. Traditional positioning technologies, especially those based on the Global Positioning System (GPS), can provide relatively accurate location information in open suburbs or highways, with an error usually between 1 and 3 meters. However, in complex urban environments (such as cities with dense high-rise buildings, tunnels, underground garages, etc.), due to signal reflection, obstruction or interference, the positioning accuracy of the GPS system will be significantly reduced or even fail, and it cannot meet the needs of autonomous driving systems for high-precision positioning.

[0232] In addition, relying on a single sensor for vehicle positioning has many limitations in practical applications. For example, when there is no signal from GPS and inertial measurement unit (IMU) for a long time, the IMU sensor is prone to cumulative errors, resulting in positioning drift.

[0233] In order to make up for the shortcomings of single positioning technology, modern assisted driving systems have begun to use multi-sensor fusion technology to improve positioning accuracy and reliability by combining data from multiple sensors (such as GPS, IMU, lidar, cameras, etc.). However, how to efficiently fuse this data and process massive environmental information in real time through algorithms remains a key challenge in the technology.

[0234] Therefore, this embodiment proposes a high-precision positioning solution that combines multi-sensor fusion technology, deep learning, and Simultaneous Localization and Mapping (SLAM) technology, which can maintain high-precision assisted driving positioning in various complex environments, which has become a technical problem that needs to be solved urgently. This embodiment aims to provide an efficient, stable, and low-latency precise positioning solution for the assisted driving system in response to the above shortcomings.

[0235] The purpose of this embodiment is to develop an assisted driving precise positioning system suitable for complex road environments and diverse climatic conditions, to improve the positioning accuracy of existing autonomous driving vehicles in complex scenarios, especially in situations where the signal is unstable or vision is limited, to ensure that the vehicle can obtain high-precision location information in real time, thereby improving the accuracy and safety of the decision-making, planning and control of the autonomous driving system.

[0236] This embodiment provides a precise positioning method and system integrating multi-sensor data fusion, deep learning algorithm and SLAM technology. The main technical solution includes the following steps: Step 1: Multi-sensor data collection.

[0237] The vehicle is equipped with multiple different types of sensors, including GPS, inertial measurement unit, lidar, millimeter-wave radar and camera, which are used to collect real-time location information, speed, acceleration, direction, environmental obstacles, etc.

[0238] Step 2: Data fusion and filtering processing.

[0239] Multi-sensor fusion technology is the key to this embodiment. By fusing the data of each sensor, more accurate vehicle location information can be obtained. Since different sensors have different data acquisition frequencies, precisions, and reliability, the synchronous fusion of multi-source data is achieved through filtering algorithms.

[0240] Among them, the extended Kalman filter: performs state estimation on nonlinear systems, solves the nonlinear problems existing in sensor data, and can correct the drift errors in GPS and IMU data in real time; the unscented Kalman filter: performs nonlinear filtering on sensor data in complex scenarios, adapts to more complex dynamic environments, and improves the accuracy and stability of data fusion through multiple iterations; the purpose of fusion processing is to integrate the data of each sensor and compensate for the defects that may exist in a single sensor. For example, when the GPS signal is weak, the IMU and lidar data are relied on for posture estimation; when the camera image is blurred, the detection data of lidar and millimeter-wave radar can be used as a substitute.

[0241] Step 3: Scene understanding and target detection.

[0242] Step 4: Real-time map construction and update based on SLAM.

[0243] In order to make up for the shortcomings of GPS signals in complex environments, this embodiment introduces simultaneous localization and mapping (SLAM) technology. SLAM uses the surrounding environment data obtained by cameras and lidar to build a local three-dimensional map of the vehicle in real time, and matches it with the pre-stored global map to ensure that the vehicle can still be accurately positioned based on the local map when the GPS signal is weak or fails.

[0244] Visual Simultaneous Localization and Mapping (VSLAM) and Laser-based Simultaneous Localization and Mapping (LSLAM) are combined. VSLAM uses the visual information obtained by the camera for positioning, while LSLAM uses laser radar to scan to obtain more accurate depth information and environmental contours. In practical applications, the combination of VSLAM and LSLAM enables the system to adapt to different environmental conditions and ensure the reliability of positioning.

[0245] Map matching and optimization: The local map constructed by SLAM technology will be matched with the global high-precision map in real time, and the continuity and accuracy of the map will be ensured through the graph optimization algorithm. Especially in environments where GPS signals fail, such as tunnels and underground parking lots, this method can effectively prevent positioning drift.

[0246] Step 5: Positioning error correction and optimization.

[0247] Through the fusion of multi-sensor data, the application of deep learning algorithms and the local map constructed by SLAM, the vehicle's positioning error can be corrected in real time to ensure that the error remains at the centimeter level; dynamic error correction, when the vehicle's position information drifts, the system will compare the real-time data obtained by the sensor with the map, and dynamically adjust the vehicle's position through the fusion processing algorithm to make it consistent with the actual road environment; intelligent optimization, the system combines deep learning models to perform intelligent analysis of the vehicle's surrounding environment, predict the movement trends of traffic participants, optimize driving paths and obstacle avoidance strategies, and further improve positioning accuracy.

[0248] Step 6: Output high-precision positioning information.

[0249] It has the following advantages: 1. Since this embodiment integrates the data of GPS, IMU, LiDAR, millimeter-wave radar and camera, high-precision positioning is achieved through algorithms such as Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF). At the same time, the weight of each sensor is dynamically adjusted according to the real-time environment, such as increasing the weight of LiDAR in low light and enhancing the weight of GPS in open areas; the adaptive weight mechanism ensures the accuracy and robustness of positioning, and effectively copes with various complex driving environments.

[0250] 2. Since this embodiment combines the advantages of VSLAM and LSLAM, they are dynamically switched and used together according to environmental conditions. Through the fusion of visual and laser data, the limitations of a single SLAM method in different environments are solved, such as the instability of visual SLAM in poor light conditions and the feature loss problem of laser SLAM in open areas.

[0251] 3. When dealing with pedestrians, vehicles and other moving objects in a dynamic environment, a dynamic object filtering method combining semantic segmentation and optical flow detection is used to remove the feature points and point cloud data of dynamic objects to avoid interfering with the precise positioning of the vehicle. At the same time, static feature points are used to correct positioning errors to ensure positioning accuracy, and stable operation can be achieved even in dynamic environments such as congestion or complex intersections.

[0252] 4. The loop detection function can identify the repeated passage of vehicles in a certain area, and achieve graph optimization by adding new constraints at the loop, thereby reducing cumulative errors. Especially in the case of long-distance driving, this patent effectively corrects map deviations through loop detection and pose graph optimization, thereby improving the accuracy and consistency of overall map construction.

[0253] 5. This embodiment provides an efficient map caching and reuse mechanism. By storing and managing local maps locally, the vehicle can directly call cached map data when passing through the same area multiple times, avoiding repeated calculations and significantly improving system efficiency. At the same time, the system supports uploading updated maps to the cloud, sharing the latest map data with other autonomous driving vehicles, realizing multi-vehicle collaborative updates, and improving the timeliness and consistency of map data.

[0254] 6. In terms of scene understanding, this patent integrates You Only Look Once (YOLO) target detection, DeepLabv3+ semantic segmentation, and behavior prediction based on Long Short-Term Memory (LSTM) to build a comprehensive environment perception system. This system can not only detect static objects and lanes, but also predict the behavior of dynamic objects (such as the future trajectory of pedestrians and vehicles), providing more accurate decision support for the autonomous driving system and ensuring safety in complex scenarios.

[0255] 7. Adopt a multi-resolution map construction method to adapt to different driving environments and map accuracy requirements; use low-resolution maps in open scenes to save computing resources, and use high-resolution maps in complex urban blocks or intersections to obtain detailed environmental information. In addition, the LSTM network is used to dynamically compensate for the accumulated error, reduce the positioning drift under long-term operation of the system, and improve the long-term stability of positioning.

[0256] Next, the modules related to the data processing involved in this embodiment are described.

[0257] refer to Fig.12 The content shown includes: a data acquisition module 1201, a data processing module 1202, a scene understanding module 1203, a positioning and map building module 1204, and a control and execution module 1205.

[0258] The data acquisition module 1201 is used to obtain all real-time information required by the vehicle during driving through multiple different types of sensors. The main sensors involved include GPS, IMU (inertial measurement unit), lidar, millimeter-wave radar and camera.

[0259] The data processing module 1202 is used to integrate data from GPS, IMU, lidar, millimeter-wave radar and camera.

[0260] The scene understanding module 1203 is used to process image data and point cloud data collected by sensors such as cameras and lidars, mainly through convolutional neural networks, to achieve real-time target detection and environment recognition.

[0261] The positioning and map construction module 1204 is used to provide high-precision positioning information even in an environment where the GPS signal is weak or invalid. This embodiment introduces simultaneous positioning and mapping (SLAM) technology to achieve accurate vehicle positioning and real-time map construction; SLAM technology can rely on lidar and cameras to generate a map of the vehicle's surroundings without a prior known map, and synchronously update the vehicle's position.

[0262] After obtaining the precise position of the vehicle, the control and execution module 1205 transmits the positioning information to the vehicle's control module for path planning, decision-making and control. The control module plans the optimal driving path based on the current position and dynamic information of the surrounding environment, and performs corresponding operations by controlling the vehicle's steering wheel, acceleration and braking.

[0263] Next, the positioning process is described.

[0264] refer to Fig.13 As shown in the content, the process may include but is not limited to the following S1301 to S1304.

[0265] S1301, data collection: including GPS, IMU, lidar, millimeter wave radar, and camera data.

[0266] The system obtains real-time information about the vehicle and its surrounding environment from a variety of sensors. It achieves high-precision environmental perception and positioning by integrating multiple sensors such as GPS, IMU, lidar, millimeter-wave radar and cameras.

[0267] S1302, data preprocessing: noise filtering, abnormal data removal, time synchronization.

[0268] Data optimization processing ensures the accuracy and stability of multi-sensor data through multiple links such as fusion, denoising, anomaly detection and error correction; including noise filtering, multi-sensor data fusion, anomaly data detection and correction, dynamic weight adjustment, and data delay optimization.

[0269] S1303, data fusion and filtering: extended Kalman filter, unscented Kalman filter, weight adjustment.

[0270] Multi-sensor data fusion and filtering adopts an improved Kalman filter and dynamic weighting algorithm to fuse multi-sensor data.

[0271] The extended Kalman filter can handle nonlinear systems and is suitable for data fusion of sensors such as GPS and IMU. In this embodiment, the extended Kalman filter is used to fuse GPS position data and IMU acceleration data, and converts sensor data into state estimation through a nonlinear function, thereby improving positioning accuracy.

[0272] Compared with the extended Kalman filter, the unscented Kalman filter is more suitable for data processing in complex scenarios, especially with better accuracy in high-dimensional nonlinear state space. The unscented Kalman filter fuses and optimizes data through unscented transformation, and is used for data fusion of lidar and millimeter-wave radar, making the system more stable in complex environments.

[0273] A dynamic weighted fusion mechanism has been introduced. Different sensors have different reliabilities under different conditions. For example, in an environment with good lighting, the weight of camera data can be higher; in rainy and snowy weather, the weight of millimeter-wave radar data is higher; through the weighted fusion system, the weight of each sensor data can be automatically adjusted according to the real-time environment to ensure the reliability of the output positioning data.

[0274] Dynamic weight adjustment,This embodiment proposes a dynamic weight adjustment strategy, which adjusts the weight of the sensor in real time according to the data quality and environmental conditions of the sensor to ensure the accuracy of the positioning data.

[0275] The weight assessment model is based on the real-time data quality, environmental conditions and historical data performance of the sensor. For example, in low-visibility weather, the weight of the camera will be relatively reduced, while the weight of the millimeter-wave radar and lidar will be increased.

[0276] The weights are adaptively adjusted. The system automatically adjusts the weights based on the confidence of multi-sensor data. The higher the confidence, the greater the weight. The confidence assessment is based on the quality of data collected in real time and the degree of deviation from the expected value. For example, when the GPS signal is stable, its weight is relatively high. If the signal fluctuates significantly, the weight automatically decreases.

[0277] On the basis of dynamic weight adjustment, the system performs weighted fusion of multi-sensor data to ensure the accuracy of the positioning results. The specific method is to perform linear weighted combination of data under different weights and obtain the optimal fused positioning data through optimization calculation.

[0278] S1304, data correction and output: dynamic object filtering, positioning error correction, and high-precision position information output.

[0279] Next, the process of data preprocessing, noise filtering, abnormal data removal, and time synchronization in S1302 is described.

[0280] Regarding noise filtering, this embodiment uses a variety of filtering techniques to perform noise removal on the data to improve the reliability of the data.

[0281] The low-pass filter is used to remove high-frequency noise in the sensor data. The Kalman filter is used for real-time data processing. The state update process of the Kalman filter includes two steps: prediction and update. Based on the vehicle's motion model, the current position and speed are predicted; the current sensor observations are used to correct the prediction results to obtain a high-precision state estimate.

[0282] Mean filtering is used to denoise lidar and camera data; median filtering is used to remove extreme outliers in lidar and millimeter-wave radar, especially when the vehicle is driving in rain, snow or foggy weather.

[0283] Abnormal data detection and correction,This embodiment adopts a variety of abnormality detection algorithms, which can identify and process abnormal data in real time, and prevent erroneous data from affecting the positioning accuracy of the system.

[0284] Threshold-based detection and time series analysis methods set a reasonable threshold range. When sensor data exceeds the normal range, the system will mark it as abnormal data and trigger the correction process. Time series analysis methods include algorithms such as moving average and autoregressive models. By analyzing the historical change trend of the data and identifying abnormal points, it can effectively detect short-term mutations and long-term drift data, thereby triggering corresponding correction strategies.

[0285] The statistical anomaly detection algorithm is based on the 3σ rule of mean and variance. For the collected sensor data, its mean and standard deviation are calculated. If a data point deviates from the mean by more than 3 times the standard deviation, the data is considered abnormal, thereby improving the robustness of the system.

[0286] For the data correction strategy, after detecting abnormal data, the system will trigger the data correction mechanism. The specific methods include: Resample the abnormal data points and obtain new sensor data to replace the abnormal points.

[0287] Use linear interpolation or spline interpolation to generate new data through surrounding data points to replace outliers.

[0288] Use historical data for retrospective analysis and revise current data through weighted average or forecasting models.

[0289] Data delay optimization: For this purpose, this embodiment designs a data delay optimization strategy to ensure that the data of all sensors are aligned at the same time, thereby ensuring the consistency of system processing.

[0290] Timestamp synchronization and data alignment: The data collected by each sensor is accompanied by timestamp information. The data is synchronized through the timestamp and data with different sampling rates are converted to the same time axis. Before multi-sensor data fusion, the system aligns the data. Data alignment uses timestamp interpolation technology to align the output of each sensor in a way that minimizes time delay.

[0291] Delay compensation and adaptive update: To avoid state lag caused by data delay, the system introduces a delay compensation mechanism, estimates data delay through a prediction model, and predicts the state based on the estimated value. For example, when IMU data lags behind GPS data, a short-term prediction can be made based on the change trend of IMU data to compensate for the impact of its delay; delay optimization requires adaptive updates based on the driving environment and sensor characteristics. The system will regularly detect changes in data delays and dynamically adjust delay compensation parameters to ensure the real-time performance of the system in different environments.

[0292] Through the above data optimization and processing flow, this embodiment can ensure the accuracy and real-time performance of multi-sensor fusion data, thereby maintaining high-precision positioning in complex autonomous driving environments, effectively responding to challenges such as severe weather and weak signal areas, and providing important technical support for the safety and stability of autonomous driving.

[0293] Next, the process of scene understanding is explained.

[0294] refer to Fig.14 The content shown may include but is not limited to the following S1401 to S1405.

[0295] S1401. Input camera image and radar point cloud data.

[0296] S1402, Object Detection: YOLO object detection, small object detection optimization, multi-object classification and bounding box.

[0297] Based on the YOLO algorithm, lanes, pedestrians, vehicles, traffic signs and other targets are detected in real time. The algorithm can simultaneously complete the positioning and classification of targets in a single forward propagation. It has the advantages of high computational efficiency and fast detection speed, and is particularly suitable for autonomous driving scenarios.

[0298] The YOLO algorithm divides the image into multiple grids, each of which is responsible for detecting objects in the area. Each grid output contains the category probability, bounding box coordinates and confidence score of the object. The main process includes: scaling and normalizing the image captured by the camera to adapt to the input requirements of the neural network; dividing the image into a grid structure, each grid predicts the category and location of one or more objects; each grid predicts the bounding box coordinates, width, height and confidence of the object; and removing redundant bounding boxes through non-maximum suppression to retain the best detection results.

[0299] Multi-target detection and classification: In complex traffic environments, vehicles need to identify multiple targets (such as pedestrians, vehicles, traffic signs, etc.) at the same time. This embodiment uses the YOLOv4 improved model with stronger multi-target detection capabilities. YOLOv4 improves the Cross Stage Partial Darknet53 (CSPDarknet53) architecture of the feature extraction part, and improves the accuracy and speed of the model through the Pyramid Pooling (Pyramid Pooling) module and cross-level connections.

[0300] Small target detection optimization: For distant or small targets, this embodiment uses a feature pyramid network (FPN) architecture to effectively improve the detection rate of small targets. FPN enhances the network's detection capability for small-scale targets by adding multi-scale feature maps in the feature extraction stage, thereby enabling the system to have higher accuracy in pedestrian detection and long-distance vehicle detection.

[0301] S1403, semantic segmentation: DeepLabv3+ segmentation lane detection, road, pedestrian, and obstacle segmentation.

[0302] Semantic segmentation can classify each pixel in the scene and assign specific category labels, such as lanes, roads, pedestrians, traffic signs, etc. Semantic segmentation plays an important role in scene understanding, helping autonomous vehicles to fully perceive road structure and environmental information. This embodiment uses the deep learning-based semantic segmentation model DeepLabv3+, combined with feature extraction and multi-scale analysis, to generate accurate pixel-level classification.

[0303] The DeepLabv3+ model consists of a backbone network and an Atrous Spatial Pyramid Pooling (ASPP) module. The ASPP module obtains rich spatial information through multi-scale feature pooling, enabling the model to recognize objects at different scales. The specific process is as follows: feature extraction, using the backbone network to extract the convolutional features of the input image; multi-scale pooling, the ASPP module pools feature maps of different scales to generate feature maps with multi-scale information; upsampling and splicing, upsampling the multi-scale feature map and splicing it with the initial feature map to form a high-resolution prediction result.

[0304] In the output stage of the semantic segmentation model, all pixels are classified into specific categories (such as roads, lanes, obstacles, etc.). Through the pixel-level classification results of semantic segmentation, the system can fully understand the spatial layout of roads and driving environment.

[0305] A specific lane line detection module is added to DeepLabv3+. Through special training, the model can effectively distinguish lane lines in a multi-lane environment, ensuring that the vehicle can accurately identify the driving lane and improve driving safety.

[0306] S1404, behavior prediction: dynamic target tracking, pedestrian / vehicle trajectory prediction, multi-target behavior prediction.

[0307] S1405. Scene understanding result output: target location and classified road and obstacle information, predicted dynamic behavior.

[0308] Behavior prediction and scenario result output, after completing basic target detection and semantic segmentation, this embodiment further introduces a scenario analysis and behavior prediction module to predict the behavior of other traffic participants and ensure that the vehicle can make reasonable predictions and avoidance.

[0309] By combining the results of target detection and semantic segmentation, a complete scene model is constructed; for example, after identifying pedestrians, the scene analysis module can determine whether the pedestrians are on the zebra crossing, whether they are crossing the road, and other information, thereby inferring potential risks.

[0310] Based on pedestrian and vehicle detection, a behavior prediction model is added. The Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) are used to predict the behavior of the detected target to infer the possible movement trajectory. For example, if a pedestrian is detected approaching a zebra crossing, the behavior prediction model can analyze its historical movement trajectory and predict whether it intends to cross the road.

[0311] In a complex traffic environment, there may be a variety of dynamic traffic participants (such as pedestrians, bicycles, other vehicles, etc.). The behavior prediction model of this embodiment is based on multi-target tracking technology. By tracking the trajectories of different targets and combining environmental semantic information, the future position of each target is predicted, thereby improving the overall prediction ability of the system.

[0312] Next, the positioning and map building process is explained.

[0313] The positioning and mapping of this embodiment combines visual SLAM (VSLAM), laser SLAM (LSLAM), map optimization and error correction technology to achieve high-precision, real-time positioning and mapping functions, combining the advantages of VSLAM and LSLAM to improve the positioning accuracy and environmental adaptability of the system.

[0314] refer to Fig.15 As shown in the content, the process may include but is not limited to the following S1501 to S1506.

[0315] S1501. Input camera image and radar point cloud data.

[0316] S1502: Feature extraction and matching.

[0317] Feature points are extracted from the image through algorithms such as Oriented FAST and Rotated BRIEF (ORB) and Scale Invariant Feature Transform (SIFT) to capture the boundaries of lane lines, obstacles and other environmental objects.

[0318] Feature matching: matching feature points in adjacent frame images to determine the vehicle's moving position in the environment.

[0319] Pose estimation, based on the matched feature points, uses the perspective n-point (PnP) algorithm to calculate the vehicle's pose change and estimate the current position.

[0320] S1503, VSLAM module: visual feature point positioning, key frame management, and 3D reconstruction.

[0321] VSLAM locates based on the visual features of the camera, manages key frames, and reconstructs the three-dimensional environment to generate a preliminary local three-dimensional map.

[0322] S1504, LSLAM module: point cloud matching and alignment, laser point cloud map construction.

[0323] Feature extraction, point cloud matching and map construction: by collecting point cloud data of the vehicle's surrounding environment, a three-dimensional point cloud reflecting the position of obstacles around the vehicle is generated, and geometric features such as lines and surfaces are extracted from the point cloud for subsequent matching and positioning; the iterative closest point (ICP) algorithm is used to align the current frame point cloud with the previous frame point cloud to calculate the vehicle's posture transformation; the local map is updated in real time through the matched and aligned point cloud data to generate a more accurate three-dimensional environment model.

[0324] S1505, data fusion and optimization: pose graph optimization, loop detection and correction, high-precision map update.

[0325] Data fusion and optimization of VSLAM and LSLAM. Using VSLAM or LSLAM alone has its limitations. VSLAM has difficulty providing stable pose estimation in environments with insufficient light or lack of texture (such as tunnels and at night), while LSLAM may lose environmental features in open spaces (such as highways); therefore, this embodiment adopts a method of combining VSLAM and LSLAM, and adaptively switches or uses them together according to environmental conditions.

[0326] S1506, positioning and map output: real-time high-precision location information, updated local map.

[0327] Switching mechanism: Through the environment perception module, the system monitors the ambient light and feature richness in real time, and adaptively switches the SLAM mode under different conditions: Good light and rich texture: VSLAM is used for positioning and map construction first, making full use of visual data. Poor light and lack of texture: switch to LSLAM mode and use lidar data to maintain positioning stability.

[0328] Fusion mechanism: In specific environments (such as urban blocks), the system can use the data of VSLAM and LSLAM at the same time to obtain more accurate pose estimation through weighted fusion: feature point fusion: match visual feature points with lidar point clouds to generate high-precision local maps; data fusion algorithm: use extended Kalman filter or particle filter (PF) to fuse the pose estimation results of VSLAM and LSLAM to reduce positioning errors.

[0329] Next, the map optimization process is described.

[0330] In order to ensure the positioning accuracy and dynamic update of the environment, this embodiment uses graph optimization technology to optimize the map; graph optimization mainly represents the vehicle posture and environmental characteristics at each moment as nodes in a graph structure, and the constraint relationships between nodes (such as the matching relationship between feature points of adjacent frames) constitute the edges of the graph, thereby reducing the positioning error by optimizing the relationship between nodes and edges.

[0331] This embodiment introduces loop closure detection technology, which is a technology for detecting vehicles passing through visited areas, which can reduce cumulative errors. When the system detects a loop, it adds new constraints to further optimize the map structure and reduce drift. In the back-end processing part of SLAM, the pose graph is used for global optimization. The nodes of the pose graph represent different positions of the vehicle, and the edges represent the pose transformation between two frames. Global map optimization is achieved by minimizing the error between nodes.

[0332] refer to Fig.16 As shown in the content, the process may include but is not limited to the following S1601 to S1605.

[0333] S1601, similarity detection: feature point matching, historical key frame benchmarking.

[0334] Similarity detection, judges the similarity between the current frame and the historical frame through feature matching, and determines the loop position.

[0335] S1602, loop confirmation: determine the loop position and generate new constraints.

[0336] Loop confirmation, when the similarity detection is successful, the system confirms the loop position, adds the relative pose between the loop node and the history node as a new constraint to the pose graph, generates a new loop constraint, and marks the position.

[0337] S1603, constraint addition: add loop constraints and update the pose graph.

[0338] The confirmed loop constraints are added to the pose graph to strengthen the relative position constraints between keyframes, adding a node for each keyframe and adding constraint edges between adjacent keyframes to reduce the cumulative error.

[0339] S1604, global optimization: pose graph optimization, nonlinear error minimization, cumulative error correction.

[0340] Global optimization: After loop closure detection is completed, all nodes are re-optimized using a graph optimization algorithm. The relative pose error between adjacent frames is calculated based on the matched feature points to eliminate the accumulated error. The pose graph is nonlinearly optimized using the Levenberg-Marquardt algorithm, and ultimately the globally optimized pose and map are obtained.

[0341] S1605, map update and output: high-precision map update, positioning error correction.

[0342] Optimized high-precision map output, including corrected positioning errors, supports vehicle navigation and environmental perception.

[0343] In a dynamic environment, error correction is a key step to improve the positioning accuracy of the system. Errors in a dynamic environment may come from interference from pedestrians, vehicles or other dynamic objects. For this reason, this embodiment introduces a variety of error correction strategies.

[0344] Dynamic object filtering: Use convolutional neural networks or optical flow methods to detect dynamic objects in the environment, and filter their point cloud or image features to avoid interference with positioning. For example, pedestrians and vehicles in the image can be separated through semantic segmentation, and the feature points of these dynamic objects can be removed during feature matching. Local map updates: Obstacles in a dynamic environment may change over time, so the system incorporates a real-time update mechanism into the local map construction process; when environmental changes are detected (such as the vehicle ahead leaving the field of view), the system automatically updates the local map to remove outdated environmental features.

[0345] Adaptive weight adjustment: In the process of map optimization, this embodiment uses an adaptive weight adjustment algorithm to adjust the weight of feature points according to their stability. For example, static features that have not changed for a long time (such as the corners of a building) are given a higher weight; dynamic features that appear briefly are reduced in weight or even eliminated, thereby improving the stability of the overall map.

[0346] In the SLAM system, positioning accuracy is crucial to the safety of autonomous driving. This embodiment further improves positioning accuracy through the following strategies: high-precision positioning data fusion. This embodiment combines the high-precision positioning data of the Global Navigation Satellite System (GNSS) and fuses the absolute position information of GNSS with the relative pose estimation of SLAM through extended Kalman filtering to improve the overall positioning accuracy of the system; multi-resolution maps. In order to adapt to different road environments, this embodiment constructs a multi-resolution map hierarchy. In open environments, low-resolution maps are used to reduce the amount of calculation; in narrow urban blocks or complex intersections, high-resolution maps are used to obtain more detailed environmental information; long-term and short-term memory error compensation. This embodiment dynamically compensates for SLAM errors through the LSTM model. LSTM can learn the error pattern of the vehicle in a specific environment, predict the positioning error at the current moment through historical data, and compensate for it, thereby reducing the accumulated error of the system.

[0347] Next, the map storage process is explained.

[0348] In order to improve the efficiency and adaptability of the system, this embodiment designs a map storage and reuse mechanism, which is particularly suitable for scenarios where vehicles pass through the same area multiple times.

[0349] refer to Fig.17 As shown in the content, the process may include but is not limited to the following S1701 to S1706.

[0350] S1701, map data generation: VSLAM / LSLAM builds and generates local high-precision maps.

[0351] S1702, map cache management: store local maps locally, set cache validity period, and determine map reuse conditions.

[0352] Local map cache: During vehicle driving, the system will cache the generated local map in local memory; when the vehicle passes through the same area again, the cached map data can be used directly to reduce repeated calculations.

[0353] S1703, map reuse: detect when the vehicle enters the cache area again, and directly call the cached map data to reduce repeated calculations.

[0354] S1704, Cloud Sharing and Collaboration: Upload updated maps to the cloud, share map data with other vehicles, and receive the latest maps from the cloud.

[0355] When multiple autonomous vehicles are driving in the same area, the system supports uploading map data to the cloud, enabling map sharing and collaborative updates among multiple vehicles, further improving map accuracy and update efficiency.

[0356] S1705. Output the latest high-precision map data containing dynamic updates and collaborative information.

[0357] After obtaining the precise location of the vehicle, the system transmits the positioning information to the vehicle's control module for path planning, decision-making and control. The control module plans the optimal driving path based on the current location and dynamic information of the surrounding environment, and performs corresponding operations by controlling the vehicle's steering wheel, acceleration and braking.

[0358] S1706, map update: Use static feature point correction to eliminate interference from dynamic objects.

[0359] The system will regularly update local maps based on the timeliness and accuracy of the maps. For areas that change frequently (such as construction sections), the system will automatically mark the validity period of the map and rebuild a new map after the expiration date; for stable areas (such as residential areas), the map data will be stored for a long time for reuse.

[0360] In summary, the positioning and map building module of this embodiment provides an efficient and reliable positioning and environment perception system for autonomous driving vehicles by combining technologies such as VSLAM, LSLAM, map optimization, dynamic error correction, high-precision data fusion, and map storage reuse. This invention can adapt to complex and changing driving environments, ensure that the vehicle always maintains high-precision positioning capabilities in dynamic environments, and significantly improve the safety and stability of the autonomous driving system.

[0361] In a second aspect, an embodiment of the present application provides a map determination device. The map determination device can be deployed in a vehicle. Fig.18 As shown in the content, the map determination device 180 may include but is not limited to: a first determination 1801, a first construction unit 1802, a second construction unit 1803, a second determination unit 1804 and a storage unit 1805.

[0362] The first determining unit 1801 is used to determine target image features of multiple target objects and target point cloud features of multiple target objects in the vehicle driving environment based on the image data and point cloud data of the vehicle during driving; A first construction unit 1802 is used to construct a three-dimensional visual map of the vehicle based on target image features of multiple target objects and first positions of the multiple target objects; the first positions are obtained based on image data; A second construction unit 1803 is used to construct a three-dimensional laser map of the vehicle based on target point cloud features of the multiple target objects and second positions of the multiple target objects; the second positions are obtained based on the point cloud data; A second determining unit 1804 is used to determine a target map for assisting vehicle driving based on at least the three-dimensional visual map and the three-dimensional laser map; The storage unit 1805 is used to store the target map, determine the stable area and the unstable area in the target map, and store the first validity period of the stable area and the second validity period of the unstable area in the target map; the first validity period is greater than the second validity period.

[0363] In some embodiments, the second determining unit 1804 is further configured to: Based on the three-dimensional visual map and the three-dimensional laser map, the target map is determined; or, based on the three-dimensional visual map and the three-dimensional laser map, the target local map is determined; based on the target local map and the global map of the vehicle, the target map is determined.

[0364] In some embodiments, the second determination unit 1804 is also used to: when the first condition is met, determine the three-dimensional visual map as the target local map; when the first condition is not met, determine the three-dimensional laser map as the target local map; wherein the first condition is used to characterize that the light intensity in the vehicle driving environment meets the light intensity threshold.

[0365] In some embodiments, the second determination unit 1804 is also used to: determine the first position of each target object in the three-dimensional visual map; determine the second position of each target object in the three-dimensional laser map; fuse the first position and the second position through a filter to obtain the target position of each target object; determine the target local map based on the target features of each target object and the target position of each target object; wherein the target features are obtained based on image features and / or point cloud features.

[0366] In some embodiments, the map determination device 180 may further include a recognition unit, which is used to: recognize image data to obtain at least one static object and at least one dynamic object; filter the recognized at least one dynamic object and determine at least one static object as a target object.

[0367] In some embodiments, when the target object includes a static object and a dynamic object, the first construction unit 1802 is also used to: respectively determine a first influence coefficient of the static object and a second influence coefficient of the dynamic object; the first influence coefficient is greater than the second influence coefficient; and construct a three-dimensional visual map based on the target image features, the first position and the first influence coefficient of the static object, and the target image features, the first position and the second influence coefficient of the dynamic object.

[0368] In some embodiments, the first determination unit 1801 is also used to: identify image data of a moving vehicle through a first network model to obtain initial feature maps of multiple target objects; perform target scale pooling processing on the initial feature maps of multiple target objects through a second network model to obtain feature maps of the target scale; and concatenate the feature map of the target scale with the initial feature map to obtain target image features of multiple target objects.

[0369] In some embodiments, the map determination device 180 may further include a processing unit, the processing unit being configured to, when the target scale includes multiple scales, the target image feature includes target image features at multiple scales; correspondingly, the three-dimensional visual map includes three-dimensional visual maps of multiple resolutions; and the target map includes target maps of multiple resolutions, perform: When the second condition is met, the vehicle is assisted in driving based on a target map of a first resolution; when the third condition is met, the vehicle is assisted in driving based on a target map of a second resolution; wherein the first resolution is lower than the second resolution, the second condition is used to characterize that the vehicle's driving environment is an open environment; and the third condition is used to characterize that the vehicle's driving environment is a narrow environment or a complex environment.

[0370] In some embodiments, the first determining unit 1801 is further configured to: Acquire initial image data and initial point cloud data of a moving vehicle; filter the initial image data and initial point cloud data to obtain first image data and first point cloud data; detect and correct abnormal data on the first image data and the first point cloud data to obtain corrected second image data and second point cloud data; time-align the second image data and the second point cloud data using a time interpolation technique to obtain aligned third image data and third point cloud data; and determine target image features of multiple target objects and target point cloud features of multiple target objects in a driving environment of the vehicle based on the third image data and the third point cloud data of the moving vehicle.

[0371] In some embodiments, the map determination device 180 may further include an adjustment unit, the adjustment unit being configured to: A first node is constructed based on the vehicle posture and environmental features of the vehicle at each moment; the environmental features are obtained based on image features and / or point cloud features; a first constraint relationship between adjacent first nodes is determined as a matching degree between environmental features of adjacent first nodes; a time graph is constructed based on the first nodes and the first constraint relationships between the first nodes; and a target map is adjusted by adjusting the first nodes and the constraint relationships between the first nodes in the time graph.

[0372] In some embodiments, the adjustment unit is also used to: construct a second node based on different positions of the vehicle; determine the edge between every two second nodes as the posture change value between every two second nodes; construct a pose graph based on the edge between the second node and the second node; adjust the second node and the edge between the second nodes in the pose graph to adjust the target map.

[0373] In some embodiments, the storage unit 1805 is further configured to: When it is detected that the second validity period of the unstable region has reached, the unstable region in the target map is reconstructed and updated.

[0374] In some embodiments, the map determination device 180 may further include a reminder unit, which is used to: Determine the predicted behaviors of multiple target objects; if the predicted behaviors of the target objects affect the driving of the vehicle, output first prompt information on the target map; the first prompt information is used to prompt that the predicted behaviors of the target objects affect the driving of the vehicle.

[0375] In a third aspect, the present application further provides a vehicle, comprising a processor and a memory, wherein the memory stores a computer program or instructions, and the computer program or instructions, when executed by the processor, implement the method provided in the first aspect.

[0376] In a fourth aspect, an embodiment of the present application provides a storage medium, that is, a computer-readable storage medium, on which a computer program or instructions are stored. When the computer program or instructions are executed by a processor, the method provided in the first aspect above is implemented.

[0377] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the method provided in the first aspect is implemented.

[0378] It should be noted here that the description of the above storage medium, device, and program product embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, device, apparatus, and program product embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0379] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0380] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0381] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0382] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0383] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0384] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories (ROM), magnetic disks or optical disks.

[0385] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0386] The above description is only an implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for determining a map, characterized in that: The method comprises: Based on the image data and the point cloud data of the vehicle during driving, respectively determining target image features of a plurality of target objects and target point cloud features of a plurality of target objects in the driving environment of the vehicle; constructing a three-dimensional visual map of the vehicle based on target image features of the plurality of target objects and first positions of the plurality of target objects; the first positions being obtained based on the image data; constructing a three-dimensional laser map of the vehicle based on target point cloud features of the multiple target objects and second positions of the multiple target objects; the second positions are obtained based on the point cloud data; Determining a target map for assisting the vehicle in driving based at least on the three-dimensional visual map and the three-dimensional laser map; The target map is stored, a stable area and an unstable area in the target map are determined, and a first validity period of the stable area and a second validity period of the unstable area are stored in the target map; the first validity period is greater than the second validity period.

2. The method according to claim 1, characterized in that The determining of a target map for assisting the vehicle driving based at least on the three-dimensional visual map and the three-dimensional laser map comprises: Determining the target map based on the three-dimensional visual map and the three-dimensional laser map; or, Determine a local map of the target based on the three-dimensional visual map and the three-dimensional laser map; The target map is determined based on the target local map and a global map of the vehicle.

3. The method according to claim 2, characterized in that The determining of the target local map based on the three-dimensional visual map and the three-dimensional laser map comprises: When the first condition is met, determining the three-dimensional visual map as the target local map; When the first condition is not met, determining the three-dimensional laser map as the target local map; The first condition is used to indicate that the light intensity in the vehicle driving environment satisfies a light intensity threshold.

4. The method according to claim 2, characterized in that: The determining of the target local map based on the three-dimensional visual map and the three-dimensional laser map comprises: Determining a first position of each target object in the three-dimensional visual map; Determining a second position of each target object in the three-dimensional laser map; Fusing the first position and the second position through a filter to obtain a target position of each target object; Based on the target feature of each of the target objects and the target position of each of the target objects, the target local map is determined; wherein the target feature is obtained based on the image feature and / or the point cloud feature.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Recognize the image data to obtain at least one static object and at least one dynamic object; The at least one identified dynamic object is filtered, and the at least one static object is determined as the target object.

6. The method according to any one of claims 1 to 4, characterized in that: In the case where the target objects include static objects and dynamic objects, constructing the three-dimensional visual map of the vehicle based on the target image features of the multiple target objects and the first positions of the multiple target objects includes: respectively determining a first influence coefficient of the static object and a second influence coefficient of the dynamic object; wherein the first influence coefficient is greater than the second influence coefficient; The three-dimensional visual map is constructed based on the target image features of the static object, the first position and the first influence coefficient, and the target image features of the dynamic object, the first position and the second influence coefficient.

7. The method according to any one of claims 1 to 4, characterized in that: Determining target image features of a plurality of target objects in the vehicle environment based on image data of the vehicle in motion, including: Recognize the image data of the vehicle in motion by using a first network model to obtain initial feature maps of multiple target objects; Performing target-scale pooling processing on the initial feature maps of the multiple target objects through a second network model to obtain target-scale feature maps; The target scale feature map is concatenated with the initial feature map to obtain target image features of the multiple target objects.

8. The method according to claim 7, characterized in that In the case where the target scale includes multiple scales, the target image feature includes target image features at multiple scales; correspondingly, the three-dimensional visual map includes three-dimensional visual maps of multiple resolutions; The target map includes target maps of multiple resolutions; The method further comprises: When the second condition is met, assisting the vehicle in driving based on the target map with the first resolution; When the third condition is met, assisting the vehicle in driving based on the target map with the second resolution; Among them, the first resolution is lower than the second resolution, the second condition is used to characterize that the driving environment of the vehicle is an open environment; and the third condition is used to characterize that the driving environment of the vehicle is a narrow environment or a complex environment.

9. The method according to any one of claims 1 to 4, characterized in that: The method of determining target image features of a plurality of target objects and target point cloud features of a plurality of target objects in the driving environment of the vehicle based on the image data and the point cloud data of the vehicle during driving comprises: Acquiring initial image data and initial point cloud data of the vehicle while it is traveling; Performing filtering processing on the initial image data and the initial point cloud data to obtain first image data and first point cloud data; Performing abnormal data detection and correction on the first image data and the first point cloud data to obtain corrected second image data and second point cloud data; Performing time alignment on the second image data and the second point cloud data by using a time interpolation technique to obtain aligned third image data and third point cloud data; Based on the third image data and the third point cloud data of the driving vehicle, target image features of multiple target objects and target point cloud features of multiple target objects in the driving environment of the vehicle are determined respectively.

10. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Constructing a first node based on the vehicle posture and environmental features of the vehicle at each moment; the environmental features are obtained based on the image features and / or point cloud features; Determining a first constraint relationship between adjacent first nodes as a matching degree between environmental features of the adjacent first nodes; constructing a time graph based on the first nodes and the first constraint relationships between the first nodes; The target map is adjusted by adjusting the first nodes in the time graph and the constraint relationship between the first nodes.

11. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: constructing a second node based on the different positions of the vehicle; Determine the edge between every two of the second nodes as the posture change value between every two of the second nodes; constructing a pose graph based on the second node and the edge between the second node; The second node and the edge between the second nodes in the pose graph are adjusted to adjust the target map.

12. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When it is detected that the second validity period of the unstable region has reached, the unstable region in the target map is reconstructed and updated.

13. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: determining predicted behaviors of the plurality of target objects; If the predicted behavior of the target object affects the driving of the vehicle, a first prompt information is output on the target map; the first prompt information is used to prompt that the predicted behavior of the target object affects the driving of the vehicle.

14. A map determination device, characterized in that: The device comprises: A first determining unit is used to determine target image features of multiple target objects and target point cloud features of multiple target objects in the driving environment of the vehicle, based on image data and point cloud data of the vehicle during driving; A first construction unit is used to construct a three-dimensional visual map of the vehicle based on target image features of the multiple target objects and first positions of the multiple target objects; the first positions are obtained based on the image data; A second construction unit is used to construct a three-dimensional laser map of the vehicle based on target point cloud features of the multiple target objects and second positions of the multiple target objects; the second position is obtained based on the point cloud data; A second determining unit, configured to determine a target map for assisting the vehicle in driving based at least on the three-dimensional visual map and the three-dimensional laser map; A storage unit is used to store the target map, determine the stable area and the unstable area in the target map, and store a first validity period of the stable area and a second validity period of the unstable area in the target map; the first validity period is greater than the second validity period.

15. A vehicle, characterized in that: The vehicle comprises a processor and a memory, wherein a computer program or instructions are stored in the memory, and when the computer program or instructions are executed by the processor, the method according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium, characterized in that: The storage medium stores a computer program or an instruction, and when the computer program or the instruction is executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images