Multi-machine collaborative simulation data generation method and system for urban large-scale scene

Abstract

The invention discloses a multi-machine collaborative simulation data generation method and system oriented to an urban large-scale scene. The method comprises the following steps: acquiring environment data through a plurality of robots carrying cameras, laser radars and IMUs, and constructing a local map based on multi-sensor fusion; triggering instance segmentation and semantic recognition by using a key frame of the laser inertial odometer, extracting semantic tags and position information of an object, and storing a high-confidence result into a database; when objects with semantic matching exist in different robot databases, calculating an optimal rotation matrix by utilizing a least square method, and splicing a local point cloud map and a grid map according to the optimal rotation matrix; loopback detection and optimization are carried out on the spliced global map, a simulation system is imported to generate a high-fidelity simulation scene, and meanwhile dynamic scene maintenance based on incremental updating is supported. The problems that in the prior art, modeling efficiency is low, map precision is poor, and simulation scene updating is lagged are solved, and the method is particularly suitable for scenes such as smart cities and automatic driving tests.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot collaborative perception and simulation technology, and in particular to a method and system for multimodal data collaborative mapping and dynamic simulation scene generation for large-scale urban environments. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, the modeling and simulation of large-scale city-level scenarios have wide-ranging application needs in fields such as smart cities, autonomous driving, and virtual reality. Existing technologies exhibit the following typical technical limitations and shortcomings.

[0003] (1) Single-robot laser inertial odometry mapping technology. Single-robot mapping was a common method for early urban map construction. Robots are usually equipped with multiple sensors to collect environmental information, among which LiDAR is widely used. LiDAR accurately calculates the distance of surrounding objects by emitting laser beams and measuring the time of their reflection, thereby obtaining point cloud data of the environment. It can quickly scan in the horizontal direction to generate point cloud data of a two-dimensional plane, providing the robot with contour information of the surrounding environment. In 2014, Zhang proposed the LOAM (LiDAR Odometry and Mapping in Real-time) framework, which is considered an important advancement in three-dimensional laser SLAM in three-dimensional reconstruction (ZHANG J, SINGH S. LOAM:Lidar Odometry and Mapping in Real-time[C / OL] / / Robotics: Science and SystemsX. Robotics: Science and Systems Foundation, 2014[2024-05-05]). This framework decomposes the complex SLAM problem into a high-frequency (10Hz) low-precision odometry thread and a low-frequency (1Hz) high-precision mapping thread, enabling a computationally efficient, low-drift, and high-precision real-time localization robot to process laser point cloud data using specific algorithms. Fast-LIO and Fast-LIO2, proposed by Zhang Fu et al. of the Mars Laboratory at the University of Hong Kong, are two representative algorithms in the field of Lidar-Inertial Odometry in recent years. Fast-LIO, as an early and efficient real-time localization and mapping solution, achieved rapid operation in resource-constrained environments by optimizing the computational flow of state estimation and map building, attracting widespread attention and application from the international research community. Subsequently, Fast-LIO2 further improved the accuracy and robustness of the algorithm based on Fast-LIO, especially significantly enhancing its adaptability in complex environments.

[0004] However, single-robot mapping is extremely inefficient because its operating range is limited to its own movement trajectory. Building maps of large-scale urban areas requires a significant amount of time. Furthermore, single-robot sensors have limited fields of view; in complex urban environments, such as city centers with towering buildings, sensors are easily obstructed, resulting in inaccurate acquisition of environmental information in some areas. Due to limitations in its data processing capabilities, positioning errors accumulate over long distances, leading to missing data or significant errors in the map, severely impacting its integrity and accuracy and failing to meet the needs of refined urban management and simulation.

[0005] (2) Traditional multi-robot collaborative mapping technology. With the increasing demands for efficiency and coverage in large-scale map construction, multi-robot collaborative mapping technology has been developed. The core idea of ​​multi-robot collaborative mapping is to allow multiple robots to collect data in different areas simultaneously, and then merge the local maps collected by each robot into a complete global map. In this process, determining the relative position and pose between the robots is crucial. Some scholars have proposed a visual feature matching method (ZOU D, TAN P. CoSLAM: Collaborative Visual SLAM in Dynamic Environments[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(2): 354-366). The robot collects images of the surrounding environment through its onboard camera, and then uses image processing technology to extract feature points in the images. By matching the feature points in the images collected by different robots, the spatial relationship between the feature points is calculated, and then the relative pose between the robots is estimated. Researchers have proposed a distributed multi-robot SLAM framework (HUANG Y, SHAN T, CHEN F, et al. DiSCo-SLAM: Distributed Scan Context-Enabled Multi-Robot LiDAR SLAM With Two-Stage Global-Local Graph Optimization[J].IEEE Robotics and Automation Letters, 2022, 7(2): 1150-1157.), which processes LiDAR point cloud data in real time, uses a two-stage global-local graph optimization procedure, and utilizes descriptors to process LiDAR point cloud data, reducing the amount of data processing. Furthermore, distance sensor-based measurement methods are also widely used. Robots continuously communicate and transmit distance data, and multiple robots can use this distance data, combined with triangulation principles, to calculate their relative positions. For example, when two robots simultaneously measure the distance to the same landmark, their relative positional relationship can be determined through geometric relationships, thus providing a basis for map merging.

[0006] However, large-scale urban environments are complex, with widespread signal interference and obstruction. The positioning signals relied upon by traditional multi-machine collaborative mapping are easily blocked by buildings, mountains, and other obstacles in densely built-up city centers, narrow streets, and underground spaces, leading to signal weakening, loss, or errors, and a significant decrease in positioning accuracy. Simultaneously, large-scale cities contain numerous similar building structures and textures, making local feature-matching-based positioning and map merging algorithms highly susceptible to misjudgments. Once misjudged, local map merging can result in misalignment, inaccurate overlap, and ultimately, global map distortion, failing to accurately reflect the true geographical information of large-scale cities and thus unable to provide reliable support for applications with stringent map accuracy requirements, such as urban simulation and autonomous driving testing.

[0007] (3) Simulation Scene Generation Technology. Simulation scene generation technology transforms a constructed map into a scene that can be used for various simulation applications. This is typically achieved using specialized 3D modeling software and related algorithms. Based on map data, 3D models of scene elements such as buildings, roads, and terrain are created. For example, using polygon modeling technology, the shape of a building is constructed by drawing a large number of polygon meshes, and then fine-tuned according to the actual building size and structure. [8] To enhance the realism of the scene, textures, lighting, and other effects are added to the model. Texture mapping technology applies real images or generated texture patterns to the surface of a 3D model, making buildings look more lifelike.

[0008] In terms of simulation scene updates, existing methods typically involve periodically re-collecting data and then updating the entire scene. In urban planning, to reflect urban development and changes, a comprehensive re-survey of the city is conducted periodically. Utilizing UAV photogrammetry... [9] Technologies such as ground-based vehicle-mounted laser scanning are used to acquire the latest urban data. Alternatively, scene modeling can be done entirely manually for use in simulation software.

[10] In terms of scene detail, there is a lack of realism and richness. Scene models often only have simple geometric shapes, lacking details such as textures and materials. For example, building walls are only filled with a single color, failing to reflect the texture of realistic materials such as bricks and glass. The addition of natural elements is also rather abrupt; vegetation is distributed irrationally, and water bodies lack a sense of flow. The lighting effects are monotonous and cannot reflect the realistic changes in light and shadow under different times and weather conditions. Effects such as the diffuse reflection of sunlight during the day and the scattering of light at night are difficult to simulate. Summary of the Invention

[0009] The purpose of this invention is to overcome the above-mentioned shortcomings of the existing technology, and to solve the problems of low efficiency of single-machine mapping, inability to meet the needs of rapid modeling of large-scale cities, insufficient accuracy of multi-machine collaborative mapping in complex environments, low realism of simulation scenes, long update cycle, and inability to reflect urban changes in real time. This invention provides a method and system for generating multi-machine collaborative simulation data for large-scale urban scenarios.

[0010] The objective of this invention is achieved through at least one of the following technical solutions.

[0011] A method for generating multi-machine collaborative simulation data for large-scale urban scenarios includes the following steps: Multi-source data on the urban environment are collected by multiple sensors mounted on multiple robots, and local maps are constructed based on the fusion of multiple sensors; Based on the keyframe triggering mechanism, instance segmentation and semantic recognition are performed on the local map to extract object semantic labels and location information, which are then stored in the database. Based on semantic matching and least squares method, the local maps of different robots are stitched together to form a global map; Loop closure detection and optimization are performed on the global map, and the data is imported into the simulation system to generate simulation scene data.

[0012] Furthermore, the multi-sensor system includes a camera, a lidar sensor, and an inertial measurement unit (IMU), and the multi-sensor fusion includes: The intrinsic and extrinsic parameters of the camera, lidar, and IMU are calibrated to obtain the pose relationship between the sensors. Input lidar and IMU data into the laser inertial odometer to generate keyframes; By fusing data from cameras, LiDAR, and IMU, local grid maps and local point cloud maps are constructed.

[0013] Furthermore, the keyframe triggering mechanism includes: When the number of keyframes generated by the laser inertial odometry reaches a preset threshold X, instance segmentation and semantic recognition are triggered. A deep learning model is used to segment data in keyframes and X frames before and after them to generate instance masks and semantic labels. The identified object is projected onto a two-dimensional plane, and its center coordinates are extracted.

[0014] Furthermore, the instance segmentation and semantic recognition also include: The identified objects are sorted by confidence level, and the top Y high-confidence objects are stored in the database. The semantically labeled text is converted into vectors using an embedding model for subsequent semantic matching.

[0015] Furthermore, the local map stitching includes: When the server detects a semantically matching object in the databases of different robots, it extracts its two-dimensional coordinates; Use the least squares method to calculate the rotation matrix so that the sum of the distances between the two sets of coordinates is minimized; The local point cloud map and the local mesh map are stitched together based on the rotation matrix; The loop closure detection and optimization include: Perform loop closure detection on the stitched global point cloud map to identify and correct pose drift. Utilizing loop closure detection results to optimize the global grid map and improve the overall performance. Figure 1 Consistency and accuracy.

[0016] Furthermore, it also includes a dynamic update step for the simulation scene: Continuous collection of environmental data by multiple robots detects changes in the scene; Re-execute data collection, map building, and optimization for the changed areas; Incremental update technology is used to integrate the updated local map into the existing simulation scene.

[0017] Furthermore, the generation of the simulation data includes: Import the optimized global mesh map into the simulation engine to build a 3D scene model; Add textures, lighting, and natural elements to generate highly realistic simulation scenes; The global point cloud map is used as prior information for relocalization in robot localization simulation.

[0018] This invention provides a multi-machine collaborative simulation data generation system for large-scale urban scenarios, implementing the aforementioned method, comprising: The data acquisition and local mapping module is used to collect data through multiple robots and multiple sensors and build local maps. The semantic extraction and storage module is used to trigger instance segmentation and semantic recognition based on keyframes, and to store semantic information; The map stitching and optimization module is used to stitch local maps based on semantic matching and perform loop closure detection and optimization. The simulation generation and update module is used to import the global map into the simulation system and supports dynamic scene updates; The data acquisition and local mapping module includes: Multi-sensor unit, including camera, LiDAR and IMU; The sensor calibration unit is used to calibrate intrinsic and extrinsic parameters and pose relationships; A laser inertial odometry unit is used to generate keyframes and local point cloud maps; Local grid reconstruction unit, used to fuse multi-sensor data to generate local grid map.

[0019] Furthermore, the semantic extraction and storage module includes: Keyframe triggering unit, used to initiate instance segmentation when the number of keyframes reaches a threshold; The instance segmentation and semantic recognition unit extracts entity masks and labels based on a deep learning model; The database management unit is used to store information about high-confidence objects and their vector representations.

[0020] Furthermore, the map stitching and optimization module includes: Semantic matching unit, used to compare object vectors in different databases; The rotation matrix calculation unit uses the least squares method to calculate the optimal rotation matrix. Loop closure detection unit is used to optimize the global point cloud map and mesh map; The simulation generation and update module includes: Scene modeling unit, used to convert grid maps into 3D models; Texture and lighting rendering unit, used to enhance the realism of the scene; Incremental update units are used to detect changes and update the simulation scene.

[0021] This invention is based on a multimodal semantic understanding and lightweight fusion architecture. It reduces the computational load through a keyframe triggering mechanism, ensures map stitching accuracy through semantic consistency, and improves scene real-time performance and consistency by combining loop closure detection and incremental updates.

[0022] Compared with the prior art, the present invention has the following advantages: (1) Improved mapping efficiency: To address the problem of low mapping efficiency of single robots, this invention aims to break the limitations of the activity range and operational capabilities of a single robot through multi-robot collaborative operation. Multiple robots simultaneously collect data in different areas of the city and work in parallel, which greatly shortens the time required to complete the construction of large-scale urban maps, meets the urgent need for rapid map generation in scenarios such as urban planning and emergency response, and ensures that basic map data support is provided for relevant decisions in the shortest possible time; (2) Improved map accuracy: To address the problem of insufficient accuracy in traditional multi-machine collaborative mapping in complex urban environments, this invention utilizes multi-sensor fusion technology, combining the advantages of various sensors to reduce the impact of signal interference and occlusion on positioning. Simultaneously, it innovatively introduces methods such as semantic text tag fusion with laser point cloud frames to enhance the accuracy of positioning and map merging in environments with similar building structures and textures. This avoids problems such as misalignment and inaccurate overlap during local map merging, enabling the constructed global map to accurately reflect the real geographic information of large-scale cities, providing a reliable map foundation for applications with extremely high map accuracy requirements, such as urban simulation and autonomous driving testing. (3) Semantic information extraction and consistency management: Semantic information extraction usually requires processing all data and is prone to information conflicts. However, this invention extracts semantic information efficiently and avoids conflicts through key frame triggering mechanism and unified semantic tag management.

[0023] (4) High realism of simulation scenes: Based on the constructed high-precision global map, this invention is committed to realizing the generation of high-quality large-scale urban simulation scenes. Using advanced 3D modeling algorithms, various scene element information in the map is accurately extracted, such as the outline, height, and structure of buildings, the direction, width, and number of lanes of roads, etc., to construct a realistic 3D scene model. At the same time, by collecting or generating texture images and combining material properties, the textures are accurately mapped onto scene elements, natural elements such as vegetation and water bodies are added, and the lighting system of the simulation software is used to adjust the lighting according to different times and weather conditions to create realistic light and shadow effects, generating simulation scenes that highly restore the real urban environment, meeting the needs of urban planning, virtual tourism and other fields for high-realism simulation scenes; (5) Strong dynamic update capability: In view of the lag in existing simulation scene update technology, this invention establishes a real-time monitoring mechanism. Multiple robots continuously collect urban environmental data and quickly detect scene changes through advanced algorithms. Once a change is detected, the entire process from data collection to map optimization is immediately re-executed for the changed area. Incremental update technology is used to quickly integrate the updated local map data into the existing simulation scene, ensuring that the simulation scene can reflect the dynamic changes of a large-scale city in real time and accurately, meeting the needs of real-time urban management and decision analysis, and providing effective technical support for the sustainable development of the city. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the overall process of a multi-machine collaborative simulation data generation method for large-scale urban scenarios in this embodiment.

[0025] Figure 2 A detailed flowchart for local map construction and semantic extraction.

[0026] Figure 3A detailed flowchart for map stitching and optimization.

[0027] Figure 4 This is a system structure block diagram in the embodiment. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. These embodiments are for illustrative purposes only and do not limit the scope of the invention.

[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0030] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0031] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] This invention achieves efficient, high-precision, and highly realistic city-level simulation data generation through multi-machine collaboration, multi-sensor fusion, semantic-driven map stitching, and dynamic updates. The technical solution is described in detail below with reference to the accompanying drawings and embodiments.

[0034] Example 1 A method for generating multi-machine collaborative simulation data for large-scale urban scenarios includes the following steps: (1) Multi-sensor data acquisition and local map construction Multiple robots are equipped with cameras, lidar, and IMUs. Through calibration, the sensor pose relationships are obtained, and the data is fused to generate local grid maps and point cloud maps.

[0035] (2) Instance segmentation and semantic information extraction Keyframe-triggered instance segmentation based on laser inertial odometry is implemented. A deep learning model extracts object masks and semantic labels, projects them onto a 2D plane, and stores high-confidence object information. By setting a keyframe threshold X (e.g., X=10), instance segmentation and semantic recognition are triggered when the number of keyframes reaches X. Mask R-CNN or a similar model is used to segment objects in the local map, generating instance masks and semantic labels. Objects are projected onto the XOY plane, and their center coordinates are extracted. The top Y objects are sorted by confidence and stored in a database. BERT or a similar embedding model is then used to convert the semantic text into vectors.

[0036] (3) Map stitching and global optimization The rotation matrix is ​​calculated using semantic matching and the least squares method, and a local map is stitched together. Loop closure detection is used to optimize the global map. When the server detects semantically matching objects in different robot databases, its two-dimensional coordinates are extracted, and the rotation matrix R is calculated using the least squares method. Based on R, a local point cloud map and a mesh map are stitched together. Loop closure detection is performed on the global point cloud map to optimize the global mesh map.

[0037] (4) Simulation scene generation and dynamic update The optimized global mesh map is imported into the simulation system, and textures, lighting, and other effects are added. Dynamic scene updates are achieved through continuous data collection and incremental updates. Alternatively, the optimized global mesh map can be imported into simulation platforms such as Unity or Unreal Engine, and elements such as textures, lighting, and vegetation can be added. Multiple robots continuously collect data to detect scene changes, and incremental update techniques are used to update the simulation scene in real time.

[0038] like Figure 1Each robot (A, B, C) is equipped with a camera, LiDAR, and inertial measurement unit (IMU). Data from these sensors, combined with intrinsic and extrinsic parameters from the robot's sensors, generates a 3D real-world reconstructed scene. The LiDAR and IMU data are input into a laser odometry system to construct a local point cloud map. Instance segmentation and semantic recognition are triggered by keyframes from the laser odometry system. A fixed value X is set. When the number of keyframes acquired by the laser odometry accumulates to X frames, local 3D real-world reconstruction is performed using visual perception data from frames n, n+1, ..., n+x. A deep learning model is then used for instance segmentation and semantic recognition within the locally reconstructed scene. Instance masks are then assigned to the segmented and recognized objects, and the semantic labels and location information for each object are obtained. The constructed 3D reconstructed scene and the identified object information are projected onto the XOY plane, and the center of all bounding boxes is taken as the coordinate point to obtain several 2D coordinates.

[0039] Each robot has a corresponding database on the server. Each identified object is bounded by a minimum regular hexagon and sorted from highest to lowest confidence level. A fixed value Y is set, and the objects with the highest confidence level (Y) for instance segmentation are grouped together and stored in the corresponding robot's database. An embedding model is used to convert the text in the database into vectors. When new vector groups are added to other robots' databases, the server matches the new vector groups with existing ones. After the semantic information of the objects is matched, the least squares method is used to obtain the rotation matrix that minimizes the sum of the distances between the two sets of 2D coordinates. After the robot's mapping task is completed, all its local maps are uploaded to the server. On the server, the local point cloud map and the local mesh map are stitched together based on the rotation matrix.

[0040] Loop closure detection is performed on the global point cloud map, and the results are used to optimize the global mesh map. The final global mesh map is imported into the simulation scene to generate simulation data for large-scale urban scenes. The global point cloud map can be used as prior map information for relocalization in the simulation.

[0041] Example 2 In some embodiments of the multi-machine collaborative simulation data generation method for large-scale urban scenarios of the present invention, each step may be implemented as follows.

[0042] 1. Local map construction based on multi-sensor fusion This embodiment deploys multiple robots, each equipped with multiple sensors including cameras, LiDAR, and inertial measurement units (IMUs), to collect multi-source data from large-scale urban scenes. Cameras are used to acquire RGB image data. LiDAR (Light Detection and Ranging) is used to acquire 3D point cloud data. The IMU is used to acquire the robot's pose and motion information. By calibrating the intrinsic and extrinsic parameters of the lidar, camera, and IMU, the pose relationships between each pair of these three components are obtained. , , and the camera's intrinsic parameter matrix Data from the lidar and IMU is input into the laser inertial odometry system to acquire keyframes for the system. K For the camera, the input of camera, LiDAR, and IMU information is used to construct a local mesh map within the large generative model. .

[0043]

[0044] During data fusion, the intrinsic and extrinsic parameters of the sensors are utilized to ensure data consistency and accuracy. The core advantage of this part lies in the fact that multi-sensor fusion significantly improves the comprehensiveness of data acquisition and the accuracy of map construction, providing a reliable foundation for subsequent semantic information extraction and map stitching.

[0045] 2. Extraction of environmental semantic information based on instance segmentation and recognition like Figure 2 After generating the local map, the next task is to extract semantic information about the environment from this data. Specifically, fixed values ​​are set... At time j, the laser inertial odometry generates a keyframe. This will trigger the process of instance segmentation and semantic recognition. This keyframe Compared to the past Keyframes The 3D local map reconstructed from all the data will then undergo instance segmentation and semantic recognition. The instance segmentation model segments each object in the local map to obtain object... Generate instance mask Meanwhile, the semantic recognition model assigns a unique semantic label to each segmented object. And extract its location information. Suppose an object instance The corresponding semantic tags are Then it can be expressed as:

[0046] To ensure semantic consistency, the semantic label of each object remains unchanged throughout the processing. After processing, the local map and the identified object information are projected onto a two-dimensional plane (XOY plane), and the center points of all object bounding boxes are taken to obtain several two-dimensional coordinates. Furthermore, objects are sorted based on the confidence score of instance segmentation, with a fixed value set. Only store confidence scores forward The object information is stored in the robot's corresponding database. The dataset D after instance segmentation can be represented as:

[0047] The key to this part is that, through the keyframe triggering mechanism and deep learning, structured semantic information in the environment is extracted efficiently while avoiding information conflicts, providing high-quality data support for map stitching and global optimization.

[0048] 3. Map stitching and simulation data generation like Figure 3 When new vector sets are added to the databases of other robots, the server matches the new vector sets with existing vector sets in the databases. Once the semantic information of objects in the two different databases matches, the system enters the map stitching stage. This embodiment calculates the rotation matrix using the least squares method, stitches together the local maps generated by different robots, and optimizes the global map using loop closure detection technology. When the server detects semantically matched objects in the databases of different robots, it uses the least squares method to calculate the rotation matrix that minimizes the sum of the distances between the two sets of two-dimensional coordinates. This involves stitching together a local point cloud map and a local mesh map. The two local maps are defined as follows: and Their two-dimensional coordinate sets are respectively and The objective function is:

[0049] During the stitching process, iterative optimization was used to reduce error accumulation and ensure stitching accuracy. Loop closure detection was then performed on the stitched global point cloud map to further optimize the map's consistency and accuracy. Finally, the optimized global mesh map was imported into the simulation scene as the basis for simulation data. Simultaneously, the global point cloud map was used as prior map information for robot relocalization within the simulation environment.

[0050] Example 3 Reference Figure 4As one embodiment, the multi-machine collaborative simulation data generation method for large-scale urban scenarios of this invention includes the following modules: a data acquisition and local mapping module, used to acquire data through multiple robots and multiple sensors and construct a local map; a semantic extraction and storage module, used to trigger instance segmentation and semantic recognition based on keyframes and store semantic information; a map stitching and optimization module, used to stitch the local map based on semantic matching and perform loop closure detection and optimization; and a simulation generation and update module, used to import the global map into the simulation system and support dynamic scene updates. The data acquisition and local mapping module includes: a multi-sensor unit, including a camera, LiDAR, and IMU; a sensor calibration unit, used to calibrate intrinsic and extrinsic parameters and pose relationships; a laser inertial odometry unit, used to generate keyframes and local point cloud maps; and a local mesh reconstruction unit, used to fuse multi-sensor data to generate a local mesh map.

[0051] The semantic extraction and storage module includes: a keyframe triggering unit, used to initiate instance segmentation when the number of keyframes reaches a threshold; an instance segmentation and semantic recognition unit, used to extract object masks and labels based on a deep learning model; and a database management unit, used to store high-confidence object information and its vector representation. The map stitching and optimization module includes: a semantic matching unit, used to compare object vectors in different databases; a rotation matrix calculation unit, used to calculate the optimal rotation matrix using the least squares method; and a loop closure detection unit, used to optimize the global point cloud map and mesh map. The simulation generation and update module includes: a scene modeling unit, used to convert the mesh map into a 3D model; a texture and lighting rendering unit, used to enhance the realism of the scene; and an incremental update unit, used to detect changes and update the simulation scene.

[0052] To verify the dynamic update capability of this invention, after the simulation scene was generated, a robot was instructed to re-patrol a portion of the area. This robot continuously collected urban environmental data and quickly detected scene changes (e.g., the movement of a car) using algorithms. Once a change was detected, the system immediately re-executed the entire process from data collection to map optimization for the affected area. Subsequently, using incremental update technology, the updated local map data was rapidly integrated into the existing simulation scene. Experimental results show that the simulation scene can respond to this change within minutes, achieving real-time reflection of urban dynamics.

[0053] Semantic information extraction typically requires processing all data and is prone to information conflicts. This invention, however, efficiently extracts semantic information and avoids conflicts through a keyframe triggering mechanism and unified semantic tag management. Local map stitching may result in error accumulation, and the application value of simulation data is limited. This invention, through map stitching and global optimization techniques, generates high-precision and consistent simulation scene data, enhancing its practical application value.

[0054] This invention employs a multi-machine collaborative and multi-sensor fusion technology, significantly improving the efficiency of data acquisition and processing, as well as the accuracy of map construction. Existing technologies typically rely on single-machine systems for data acquisition, resulting in limited coverage and often using a single sensor (such as LiDAR or a camera), leading to a single data source and susceptibility to environmental noise or sensor limitations. In contrast, this invention deploys multiple robots to collect data in parallel, combining data from cameras, LiDAR, and inertial measurement units (IMUs) to generate high-precision local grid maps and local point cloud maps using multi-sensor fusion technology. This multi-machine collaborative and multi-sensor fusion approach not only expands the data acquisition coverage but also improves the map's detail and robustness, providing a more reliable foundation for subsequent semantic information extraction and map stitching.

[0055] Secondly, this invention efficiently extracts and stores environmental semantic information through a keyframe triggering mechanism and deep learning technology, significantly reducing computational resource consumption. Existing technologies typically require instance segmentation and semantic recognition of all collected data, resulting in high computational costs and the potential for the same object to be assigned different semantic labels in different frames or by different robots, leading to information conflicts. In contrast, this invention generates keyframes using laser inertial odometry, performing instance segmentation and semantic recognition only on the keyframe and a certain range before and after it, drastically reducing computational load. Simultaneously, a unified semantic labeling mechanism ensures consistent semantic information for each object throughout the processing. Furthermore, a confidence-based data filtering mechanism stores only the object information with the highest confidence scores (Y), further improving data quality and usability. These techniques not only optimize the allocation of computational resources but also avoid information conflicts, providing high-quality structured data for subsequent map stitching and optimization.

[0056] This invention generates high-precision and consistent simulation scene data through map stitching and global optimization techniques, significantly improving the practicality and application value of simulation data. In existing technologies, local map stitching may result in error accumulation, leading to inconsistencies in the global map, and the generated simulation data is typically only used for scene visualization, lacking practical application value. This invention optimizes the accuracy of local map stitching by calculating the rotation matrix using the least squares method, and further optimizes the global map using loop closure detection technology, effectively reducing error accumulation and improving the accuracy of the global map. The final generated global mesh map can be directly used for simulation scene construction, while the global point cloud map can serve as prior map information for relocalization, enhancing the practicality and application value of the simulation data. This global map optimization technique not only improves scene consistency but also provides reliable technical support for the simulation and relocalization of large-scale urban scenes.

[0057] This method can be applied to scenarios such as smart city management, autonomous driving simulation testing, and virtual tourism. For example, in autonomous driving testing, the high-precision simulation scenarios generated by this invention can be used to simulate complex traffic environments, improving test safety and coverage.

[0058] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail or in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0059] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0060] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A method for generating multi-machine collaborative simulation data for large-scale urban scenarios, characterized in that, Includes the following steps: Multi-source data on the urban environment are collected by multiple sensors mounted on multiple robots, and local maps are constructed based on the fusion of multiple sensors; Based on the keyframe triggering mechanism, instance segmentation and semantic recognition are performed on the local map to extract object semantic labels and location information, which are then stored in the database. Based on semantic matching and least squares method, the local maps of different robots are stitched together to form a global map; Loop closure detection and optimization are performed on the global map, and the data is imported into the simulation system to generate simulation scene data.

2. The method according to claim 1, characterized in that, The multi-sensor system includes a camera, lidar, and an inertial measurement unit (IMU), and the multi-sensor fusion includes: The intrinsic and extrinsic parameters of the camera, lidar, and IMU are calibrated to obtain the pose relationship between the sensors. Input lidar and IMU data into the laser inertial odometer to generate keyframes; By fusing data from cameras, LiDAR, and IMU, local grid maps and local point cloud maps are constructed.

3. The method according to claim 2, characterized in that, The keyframe triggering mechanism includes: When the number of keyframes generated by the laser inertial odometry reaches a preset threshold X, instance segmentation and semantic recognition are triggered. A deep learning model is used to segment data in keyframes and X frames before and after them to generate instance masks and semantic labels. The identified object is projected onto a two-dimensional plane, and its center coordinates are extracted.

4. The method according to claim 3, characterized in that, The instance segmentation and semantic recognition also include: The identified objects are sorted by confidence level, and the top Y high-confidence objects are stored in the database. The semantically labeled text is converted into vectors using an embedding model for subsequent semantic matching.

5. The method according to claim 1, characterized in that, The local map stitching includes: When the server detects a semantically matching object in the databases of different robots, it extracts its two-dimensional coordinates; Use the least squares method to calculate the rotation matrix so that the sum of the distances between the two sets of coordinates is minimized; The local point cloud map and the local mesh map are stitched together based on the rotation matrix; The loop closure detection and optimization include: Perform loop closure detection on the stitched global point cloud map to identify and correct pose drift. Utilize loop closure detection results to optimize the global grid map, improving map consistency and accuracy.

6. The method according to claim 1, characterized in that, It also includes the dynamic update step of the simulation scene: Continuous collection of environmental data by multiple robots detects changes in the scene; Re-execute data collection, map building, and optimization for the changed areas; Incremental update technology is used to integrate the updated local map into the existing simulation scene.

7. The method according to claim 1, characterized in that, The simulation data generation includes: Import the optimized global mesh map into the simulation engine to build a 3D scene model; Add textures, lighting, and natural elements to generate highly realistic simulation scenes; The global point cloud map is used as prior information for relocalization in robot localization simulation.

8. A multi-machine collaborative simulation data generation system for large-scale urban scenarios, used to implement the multi-machine collaborative simulation data generation method for large-scale urban scenarios as described in claim 1, characterized in that, include: The data acquisition and local mapping module is used to collect data through multiple robots and multiple sensors and build local maps. The semantic extraction and storage module is used to trigger instance segmentation and semantic recognition based on keyframes, and to store semantic information; The map stitching and optimization module is used to stitch local maps based on semantic matching and perform loop closure detection and optimization. The simulation generation and update module is used to import the global map into the simulation system and supports dynamic scene updates; The data acquisition and local mapping module includes: Multi-sensor unit, including camera, LiDAR and IMU; The sensor calibration unit is used to calibrate intrinsic and extrinsic parameters and pose relationships; A laser inertial odometry unit is used to generate keyframes and local point cloud maps; Local grid reconstruction unit, used to fuse multi-sensor data to generate local grid map.

9. The system according to claim 8, characterized in that, The semantic extraction and storage module includes: Keyframe triggering unit, used to initiate instance segmentation when the number of keyframes reaches a threshold; The instance segmentation and semantic recognition unit extracts entity masks and labels based on a deep learning model; The database management unit is used to store information about high-confidence objects and their vector representations.

10. The system according to claim 9, characterized in that, The map stitching and optimization module includes: Semantic matching unit, used to compare object vectors in different databases; The rotation matrix calculation unit uses the least squares method to calculate the optimal rotation matrix. Loop closure detection unit is used to optimize the global point cloud map and mesh map; The simulation generation and update module includes: Scene modeling unit, used to convert grid maps into 3D models; Texture and lighting rendering unit, used to enhance the realism of the scene; Incremental update units are used to detect changes and update the simulation scene.

Images