Method, apparatus, medium, and vehicle for localization and map building synchronization
By employing instance segmentation and joint optimization techniques, the problem of inaccurate object motion trajectories in dynamic scenes was solved, enabling accurate positioning and map construction of the vehicle itself and its surrounding environment, thereby improving the intelligent driving service experience and safety of autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHERY AUTOMOBILE CO LTD
- Filing Date
- 2023-03-30
- Publication Date
- 2026-06-02
Smart Images
Figure CN116358586B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous driving technology, and in particular relates to a method, device, medium and vehicle for synchronizing localization and map building. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Currently, most existing object-level dynamic SLAM systems employ classic map representations already used in static SLAM systems, such as point clouds, surfels, or volumetric maps. This results in partial or incomplete object mapping observable information being incorporated into the object model, while information about invisible parts cannot be filled in unless an object or sensor is actively moving.
[0004] Instead of reconstructing objects from scratch, some approaches explore learning-based category-level object shape priors and construct object-level mappings based on these priors. Object geometry and pose are typically optimized through differentiable rendering; however, most systems are only suitable for static scenes. Furthermore, while capable of generating complete object geometry, object shape priors cannot capture complex geometric details, acting as a bottleneck; their implicit representations can only interpolate shapes within the training dataset. Combined with close-up image comparisons, such as photometric or ICP residuals, these measurements are inconsistent with previous object data, inevitably leading to inaccurate estimations of object trajectories. This results in unstable system performance, hindering simultaneous localization and mapping of the autonomous vehicle and surrounding vehicles, thus reducing the driving experience and safety for autonomous driving users. Summary of the Invention
[0005] To address the technical problems mentioned above, this invention provides a method, apparatus, medium, and vehicle for simultaneous positioning and map building. These features high receiving sensitivity, fast data transmission and processing speed, and stable and reliable performance. They enable simultaneous positioning and map building of both the vehicle itself and surrounding vehicles, and provide lateral and longitudinal system control services for the vehicle based on the detection results, offering users a more intelligent driving experience and enhanced safety.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] The first aspect of the present invention provides a method for synchronizing positioning and map building.
[0008] A method for synchronizing localization and map building, comprising:
[0009] Acquire a depth image of the environment surrounding the vehicle, perform instance segmentation on the depth image of the environment surrounding the vehicle, and simultaneously track the pose of the image acquisition device and the background region to determine the model of each visible object.
[0010] The model and geometry of each visible object are tracked to jointly optimize the geometry and pose of each visible object. The joint optimization results are then integrated with the overlap rate data of instance segmentation to determine the initialization of new objects.
[0011] Based on the tracking of the newly initialized object, the geometry and pose of the newly initialized object are jointly optimized again. The results of this joint optimization are then integrated with the instance segmentation data to achieve the highest overlap rate. Finally, the object with the most accurate geometry and pose is integrated and simultaneously located and displayed in the instantiated map.
[0012] A second aspect of the present invention provides an apparatus for synchronizing positioning and map building.
[0013] A method for synchronizing localization and map building, comprising:
[0014] The instance segmentation and object determination module is used to acquire a depth image of the environment surrounding the vehicle, perform instance segmentation on the depth image of the environment surrounding the vehicle, and simultaneously track the pose background region of the image acquisition device to determine the model of each visible object.
[0015] The initialization new object determination module is used to track the model and geometry of each visible object to jointly optimize the geometry and pose of each visible object, and integrate the joint optimization results with the instance segmentation overlap rate data to determine the initialization new object;
[0016] The synchronous positioning map building module is used to track the determined new initial object, jointly optimize the geometry and pose of the new initial object again, integrate the joint optimization result with the instance segmentation data for a second overlap rate, and finally integrate the object with the most accurate geometry and pose, and synchronously position and display it in the instantiated map.
[0017] A third aspect of the present invention provides a computer-readable storage medium.
[0018] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method for synchronizing localization and map building as described above.
[0019] A fourth aspect of the present invention provides a vehicle.
[0020] A vehicle includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method for synchronizing positioning and map building as described above.
[0021] Compared with the prior art, the beneficial effects of the present invention are:
[0022] The method implemented in this invention is relatively advanced, with high receiving sensitivity, fast data transmission and processing speed, and stable and reliable performance. The system can simultaneously locate and map the vehicle itself and surrounding targets, and implement lateral and longitudinal system control services for the vehicle based on the detection results, providing users with a more intelligent driving service experience and safety assurance.
[0023] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0025] Figure 1 This is a schematic diagram of the SLAM principle framework;
[0026] Figure 2 This is a flowchart of the method for synchronizing positioning and map construction according to an embodiment of the present invention. Detailed Implementation
[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0028] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0029] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0030] Example 1
[0031] This embodiment provides a vehicle simultaneous localization and mapping scheme. The scheme is relatively advanced, featuring high receiving sensitivity, fast data transmission and processing speed, and stable and reliable performance. The system can simultaneously locate the vehicle itself and surrounding targets, and build maps. Based on the detection results, it implements lateral and longitudinal system control services for the vehicle, providing users with a more intelligent driving experience and enhanced safety.
[0032] To solve the aforementioned technical problems, the present invention adopts the following solution:
[0033] A method for synchronizing localization and map building, comprising:
[0034] Acquire a depth image of the environment surrounding the vehicle, perform instance segmentation on the depth image of the environment surrounding the vehicle, and simultaneously track the pose of the image acquisition device and the background region to determine the model of each visible object.
[0035] The model and geometry of each visible object are tracked to jointly optimize the geometry and pose of each visible object. The joint optimization results are then integrated with the overlap rate data of instance segmentation to determine the initialization of new objects.
[0036] Based on the tracking of the newly initialized object, the geometry and pose of the newly initialized object are jointly optimized again. The results of this joint optimization are then integrated with the instance segmentation data to achieve the highest overlap rate. Finally, the object with the most accurate geometry and pose is integrated and simultaneously located and displayed in the instantiated map.
[0037] This paper proposes a novel object-level mapping system capable of simultaneously segmenting, tracking, and reconstructing objects in dynamic scenes. RGB-D cameras are deployed on vehicles for simultaneous localization and mapping. The acquired signals are transmitted to the system for processing and are displayed in real-time on the host domain controller, allowing the driver to view their position and surrounding environment. The system trains and processes the acquired signals, transmitting the results to the control system for control decisions. Vehicles can further predict and reconstruct their full geometry by adjusting depth input and category-level shape priors, aiming to achieve better object geometry and improved target reconstruction and tracking accuracy. For each incoming RGB-D frame, we perform instance segmentation to detect objects and establish data associations between detections. Existing object mappings are used to create new object mappings for each detection, resulting in a high-performance detection. For each matched object, we collectively optimize its pose and latent geometric representation, using geometric residuals and differential mapping to render the residuals, prioritizing shape and completing the geometry, providing users with a more intelligent autonomous driving service experience and enhanced safety.
[0038] Here, pose refers to the detected posture of an object, with line segments representing key branches. Geometry refers to the overall shape of the object and the shapes of its parts. Residual algorithms are used to reduce the error compared to the actual object and approximate the actual object's pose and geometry.
[0039] Simultaneous Localization and Mapping (SLAM) research aims to simultaneously estimate the geometrically unknown environment and the robot's pose within it. This can only be achieved using data from onboard sensors. It has rapidly evolved from sparse SLAM to dense SLAM, and recently has entered the realm of semantic object-level SLAM. Figure 1 A schematic diagram of the SLAM principle is presented. This rapidly evolving research enables many autonomous driving applications; however, most SLAM research still assumes that points in a static scene maintain a constant position in global spatial coordinates in the 3D world. Any information that violates this assumption, such as moving objects in the environment, is considered an outlier and intentionally ignored in tracking and mapping steps. However, this setup can only handle a small number of dynamic elements, excluding itself from many real-world scenarios. Applications, especially those involving human environments, are constantly changing, and robust SLAM systems are therefore desirable for handling highly dynamic environments.
[0040] Reference Figure 2 Each input RGB-D image is processed to perform instance segmentation, tracking the camera pose and background region, excluding mask regions and moving objects. Object geometry is represented by two nodes sharing the object pose: a front node and a back node. Latent conditional encoding can be used to represent the category-level shape of the front node. During tracking of each visible object model and geometry, when object tracking is lost, the previous node is used to initialize the object pose and reposition the object model shape. A portion of the encoded result from the back node and its associated weight volume are converted into a feature volume. The scene is then reconstructed using the repositioned object model shape and the converted feature volume. Thus, a complete occupancy domain can be predicted at any given 3D location, using shape to complete network construction.
[0041] This embodiment also incorporates a latent conditional encoding to optimize the latent space and generate novel shapes. The shape completion network shares a similar architecture but also employs additional input confidence weights and instance-level latent codes. An efficient axis-aligned bounding box ray intersection test is performed to find all visible, current viewpoint, and existing object models in the rendering, along with the object mask for each visible model. Detections on the current frame are overridden, jointly (IoU), and the rendered model mask is computed to build an association between the current frame detections and existing object models. Then, each object model is jointly optimized to complete its pose and geometry. Using the estimated pose for both the camera and the object, new depth measurements are fused into an object model and a complete shape geometry, which can be predicted by tuning the fused model. New objects are created by initializing for mismatched detections; these initial object poses use object prior models. Specifically, an RBG-D camera detection system is used to find all nearby vehicles. For each vehicle, a motion planner is used at each high-level instruction, and all motion plans above the high-level instructions are used to check for collisions with the ego vehicle's motion plan, ensuring the ego vehicle avoids collisions.
[0042] Example 2
[0043] This embodiment provides a device for synchronizing positioning and map building, which includes:
[0044] The instance segmentation and object determination module is used to acquire a depth image of the environment surrounding the vehicle, perform instance segmentation on the depth image of the environment surrounding the vehicle, and simultaneously track the pose background region of the image acquisition device to determine the model of each visible object.
[0045] The initialization new object determination module is used to track the model and geometry of each visible object to jointly optimize the geometry and pose of each visible object, and integrate the joint optimization results with the instance segmentation overlap rate data to determine the initialization new object;
[0046] The synchronous positioning map building module is used to track the determined new initial object, jointly optimize the geometry and pose of the new initial object again, integrate the joint optimization result with the instance segmentation data for a second overlap rate, and finally integrate the object with the most accurate geometry and pose, and synchronously position and display it in the instantiated map.
[0047] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.
[0048] Example 3
[0049] This embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for synchronizing positioning and map building as described above.
[0050] Example 4
[0051] This embodiment provides a vehicle including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the positioning and map building synchronization method described above.
[0052] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for synchronizing positioning and map building, characterized in that, include: Acquire a depth image of the environment surrounding the vehicle, perform instance segmentation on the depth image of the environment surrounding the vehicle, and simultaneously track the pose of the image acquisition device and the background region to determine the model of each visible object. The model and geometry of each visible object are tracked to jointly optimize the geometry and pose of each visible object. The joint optimization results are then integrated with the overlap rate data of instance segmentation to determine the initialization of new objects. The geometry of each visible object consists of two nodes: a shared object pose before node and a shared object pose after node. During the tracking of each visible object model and geometry, when object tracking is lost, the previous node is used to initialize the object pose and reposition the object model shape. A portion of the encoded result from the subsequent node is selected and its associated weight volume is converted into a feature volume. The scene is reconstructed using the repositioned object model shape and the converted feature volume. Together, the pose and potential geometric representation of the detected object are optimized. The geometric residual and differential are used to draw the residual to approximate the actual object pose and geometry. Based on the tracking of the newly initialized object, the geometry and pose of the newly initialized object are jointly optimized again. The results of this joint optimization are then integrated with the instance segmentation data to achieve the highest overlap rate. Finally, the object with the most accurate geometry and pose is integrated and simultaneously located and displayed in the instantiated map.
2. The method for synchronizing positioning and map building as described in claim 1, characterized in that, The shared object pose front node represents its category-level shape.
3. The method for synchronizing positioning and map building as described in claim 1, characterized in that, In the process of jointly optimizing the geometry and pose of each visible object, a correlation is built between the current frame detection data and the existing object model.
4. An apparatus for synchronizing positioning and map building using the method described in claim 1, characterized in that, include: The instance segmentation and object determination module is used to acquire a depth image of the environment surrounding the vehicle, perform instance segmentation on the depth image of the environment surrounding the vehicle, and simultaneously track the pose background region of the image acquisition device to determine the model of each visible object. The initialization new object determination module is used to track the model and geometry of each visible object to jointly optimize the geometry and pose of each visible object, and integrate the joint optimization results with the instance segmentation overlap rate data to determine the initialization new object; The synchronous positioning map building module is used to track the determined new initial object, jointly optimize the geometry and pose of the new initial object again, integrate the joint optimization result with the instance segmentation data for a second overlap rate, and finally integrate the object with the most accurate geometry and pose, and synchronously position and display it in the instantiated map.
5. The device for synchronizing positioning and map building as described in claim 4, characterized in that, The geometry of each visible object consists of two nodes: a shared object pose before node and a shared object pose after node.
6. The apparatus for synchronizing positioning and map building as described in claim 5, wherein the shared object pose preceding node represents its category-level shape.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method for synchronizing positioning and map building as described in any one of claims 1-3.
8. A vehicle, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method for synchronizing positioning and map building as described in any one of claims 1-3.