A three-dimensional semantic reconstruction method, device, equipment and storage medium for multi-agent perception fusion in a park

Abstract

The present application relates to the field of park management and autonomous driving technology, and discloses a three-dimensional semantic reconstruction method, device, equipment and storage medium for multi-agent perception fusion in a park, wherein the method includes: time synchronization and initial positioning, semantic segmentation image, outlier removal, projection of lidar points onto the semantically segmented image, interpolation to obtain dense depth image, multi-agent network data synchronization, construction of multi-agent collaborative map, redundant filtering and three-dimensional reconstruction, and network sharing and real-time map update and maintenance. The present application combines lidar, camera and IMU data, and utilizes advanced SLAM algorithms and deep learning technology to achieve high-precision multi-sensor data fusion and precise alignment. This not only improves the accuracy of three-dimensional semantic reconstruction, but also enhances the reliability of the system, enabling it to work stably under various environmental conditions.

Description

Technical Field

[0001] The present application relates to the field of park management and autonomous driving technology, and in particular to a three-dimensional semantic reconstruction method, device, equipment and storage medium for multi-agent perception fusion in a park. Background Art

[0002] In today's campus management and autonomous driving fields, the collaborative work of multi-agent systems is crucial for achieving efficient and accurate 3D semantic reconstruction. However, existing methods face numerous challenges in achieving this goal. First, traditional multi-agent perception systems often rely on data from a single sensor, such as lidar or a camera, which limits the system's perception range and accuracy. Furthermore, the accuracy of semantic segmentation and point cloud registration during the data fusion process directly impacts the quality of the final reconstruction.

[0003] When processing LiDAR point cloud data, existing technologies often lack effective methods for outlier removal and deep image interpolation, which can lead to errors and discontinuities in the reconstructed 3D models. Furthermore, network synchronization and data sharing between multiple agents often suffer from latency and efficiency issues. Furthermore, existing map registration methods are often inefficient when processing large amounts of data and difficult to update and maintain in real time. Summary of the Invention

[0004] In order to solve the above problems, the present application provides a three-dimensional semantic reconstruction method, device, equipment and storage medium for multi-agent perception fusion of a park, so as to solve the problems existing in the existing technology in multi-agent perception fusion, image and point cloud registration, and three-dimensional semantic reconstruction, and provide an efficient and accurate solution for the fields of park management and autonomous driving.

[0005] In order to achieve the above objectives, the technical solutions adopted in this application are as follows:

[0006] In a first aspect, the present application provides a method for 3D semantic reconstruction based on multi-agent perception fusion in a park, the method comprising:

[0007] Time synchronization and initial positioning: Initialize a global coordinate system and global map, and use the sensor fusion of lidar and camera in combination with the SLAM algorithm to achieve the initial positioning of the intelligent agent in the global map at the initial position in the charging room;

[0008] Semantic segmentation of images: Perform semantic segmentation on camera-captured images based on deep learning models to extract semantic information of different objects and scene elements;

[0009] Outlier removal: Identify and remove outliers in the LiDAR point cloud by setting a point cloud processing algorithm;

[0010] LiDAR point projection to the semantically segmented image: Using the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected to the corresponding position of the semantically segmented image to obtain a sparse depth map corresponding to the monocular image;

[0011] Interpolation to obtain dense depth image: Use the nearest neighbor interpolation method to upsample the sparse depth map;

[0012] Multi-agent network data synchronization: When each agent collects data, it timestamps the data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp;

[0013] Constructing a multi-agent collaborative map: The central computer uses an iterative closest point algorithm to initially align the local maps of each agent with the global map. Based on the initially aligned global map, a global optimization algorithm is used to further optimize the alignment results of each local map. The optimized local maps are then integrated into the global map, and the global map is updated in real time.

[0014] Redundancy filtering and 3D reconstruction: Redundancy filtering is performed by setting thresholds and clustering algorithms, and 3D reconstruction is performed;

[0015] Network sharing and real-time map update and maintenance: The central computer regularly receives sensor data from each intelligent agent and uses the sensor data of each intelligent agent to update the global map. The updated map data is sent back to each intelligent agent.

[0016] In a second aspect, the present application provides a three-dimensional semantic reconstruction device for multi-agent perception fusion in a park, the device comprising:

[0017] The synchronous positioning module is configured for time synchronization and initial positioning: it initializes a global coordinate system and a global map, and uses the sensor fusion of lidar and camera in combination with the SLAM algorithm to achieve the initial positioning of the intelligent agent in the global map at the initial position in the charging room;

[0018] The image segmentation module is configured to semantically segment images: it performs semantic segmentation on the images captured by the camera based on a deep learning model to extract semantic information of different objects and scene elements;

[0019] The outlier removal module is configured to identify and remove outliers in the lidar point cloud by setting a point cloud processing algorithm;

[0020] The image projection module is configured to project the LiDAR points onto the semantically segmented image: utilizing the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected onto the corresponding positions in the semantically segmented image to obtain a sparse depth map corresponding to the monocular image;

[0021] The image interpolation module is configured to interpolate and obtain a dense depth image: the sparse depth image is upsampled using the nearest neighbor interpolation method;

[0022] The data synchronization module is configured to synchronize data in a multi-agent network: when each agent collects data, it timestamps the collected data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp;

[0023] The map construction module is configured to construct a multi-agent collaborative map: the central computer uses an iterative closest point algorithm to initially align the local maps of each agent with the global map. Based on the initially aligned global map, a global optimization algorithm is used to further optimize the alignment results of each local map. The optimized local maps are then integrated into the global map, and the global map is updated in real time.

[0024] The 3D reconstruction module is configured for redundancy filtering and 3D reconstruction: redundancy filtering is performed by setting a threshold and a clustering algorithm, and 3D reconstruction is performed;

[0025] The update and maintenance module is configured for network sharing and real-time map update and maintenance: the central computer regularly receives sensor data from each intelligent agent, and uses the sensor data of each intelligent agent to update the global map, and the updated map data is sent back to each intelligent agent.

[0026] In a third aspect, the present application provides a three-dimensional semantic reconstruction device for multi-agent perception fusion in a campus, and the system includes: a memory for storing a computer program; and a processor for executing the computer program to implement the method described above.

[0027] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, executes the method described above.

[0028] This application has at least the following beneficial effects:

[0029] 1. Improved Fusion Accuracy and Reliability: This application combines LiDAR, camera, and IMU data, leveraging advanced SLAM algorithms and deep learning techniques to achieve high-precision multi-sensor data fusion and precise registration. This not only improves the accuracy of 3D semantic reconstruction but also enhances the reliability of the system, enabling it to operate stably in a variety of environmental conditions.

[0030] 2. Optimizing Data Processing and Reconstruction Quality: Through an effective outlier removal algorithm and depth image optimization techniques that overlay semantic information, this application can identify and remove outliers in LiDAR data. Furthermore, improved interpolation techniques are used to generate high-quality dense depth images. These techniques significantly improve the quality of 3D reconstruction results and reduce errors.

[0031] 3. Enhanced system real-time performance and adaptability: This application incorporates a multi-agent network synchronization mechanism and real-time map update and maintenance technology, enabling rapid data sharing and dynamic map updates. This enables the system to promptly respond to environmental changes, improving the real-time performance and adaptability of park management and autonomous driving, and ensuring the safety and efficiency of park inspections. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a three-dimensional semantic reconstruction method of multi-agent perception fusion in a park according to an embodiment of the present application is shown.

[0033] Figure 2 A schematic diagram of a charging chamber and a global coordinate system center according to an embodiment of the present application is shown.

[0034] Figure 3 A schematic diagram of the intelligent body structure according to an embodiment of the present application is shown.

[0035] Figure 4 The figure shows segmentation accuracy data of different rounds according to an embodiment of the present application.

[0036] Figure 5 A schematic diagram of a good points selection method according to an embodiment of the present application is shown.

[0037] Figure 6 A flowchart of projecting lidar point cloud data onto a camera image plane according to an embodiment of the present application is shown.

[0038] Figure 7 A schematic diagram of laser radar point projection onto an image according to an embodiment of the present application is shown.

[0039] Figure 8 A diagram showing the significance of a campus inspection according to an embodiment of the present application is shown.

[0040] Figure 9 A structural diagram of a three-dimensional semantic reconstruction device for multi-agent perception fusion in a park according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0042] The specific implementation of the present application is further described in detail below with reference to the accompanying drawings and examples.

[0043] In order to solve the problems existing in the existing technology in multi-agent perception fusion, image and point cloud registration, and three-dimensional semantic reconstruction, an efficient and accurate solution is provided for the field of park management and autonomous driving. The embodiment of the present application provides a three-dimensional semantic reconstruction method for multi-agent perception fusion in a park, which realizes accurate data synchronization between multiple agents by establishing a global coordinate system and time synchronization. Using semantic segmentation technology, the method can extract accurate semantic information from the image and combine it with the lidar point cloud data to generate high-quality dense depth images through projection and innovative interpolation technology. In addition, the method also includes an efficient outlier removal algorithm to ensure the accuracy of the point cloud data.

[0044] Regarding multi-agent network synchronization, this method proposes an innovative technique based on network-shared data and map registration. This technique effectively handles data exchange between multiple agents, enabling fast and accurate map construction. By combining redundant filtering with 3D reconstruction, this method further improves reconstruction efficiency while ensuring the accuracy and robustness of the reconstruction results. This enables the 3D semantic map to dynamically reflect environmental changes within the park, providing strong support for park management and autonomous driving.

[0045] Specifically, this method can effectively solve the problems of multi-agent network synchronization and map misalignment through the sensors installed on the agent (AGV) and the campus network, improve the robustness, adaptability and real-time performance of the system, and ensure the efficient collaborative work of the multi-agent system in a complex dynamic environment. Figure 1 , which is a flow chart of a three-dimensional semantic reconstruction method based on multi-agent perception fusion in a park. The three-dimensional semantic reconstruction method based on multi-agent perception fusion in a park includes the following steps S1 to S9, which are introduced in detail as follows.

[0046] S1: Time synchronization and initial positioning

[0047] Initialize a global coordinate system and global map through the central computer. Initial position and posture information in the charging room. Utilize the sensor fusion technology of LiDAR and camera, combined with SLAM (Simultaneous Localization and Mapping) algorithm, to achieve the initial positioning of the intelligent body in the global map, such as Figure 2 Schematic diagram of the charging room and the center of the global coordinate system. In the figure, the central computer 202 is set in the charging room 201. There are multiple charging piles 204 in the charging room 201. The charging piles 204 are used to charge the intelligent body 203. The structure of the intelligent body 203 is as follows: Figure 3 As shown, the intelligent body 203 includes a robot body 307, on which an IMU 301, a laser radar 302, a camera 304 and a network transmission module 306 are provided; wherein 303 is the scanning area of ​​the laser radar 302, and 305 is the viewing angle of the camera.

[0048] More specifically: In an embodiment of the present application, a unified coordinate system is first initialized by a central computer. Each intelligent body (AGV) waits at the charging pile to which it belongs for the central computer to issue an inspection work order. After receiving the work order, the intelligent body uses the onboard laser radar and camera to obtain environmental features. This patent performs precise positioning through an improved ORBL-SLAM algorithm. The ORBL-SLAM algorithm is an open source navigation and positioning algorithm based on corner detection and fusion of radar point clouds developed based on the ORB-SLAM algorithm, which can update the position and posture of the intelligent body in real time. ORB-SLAM only supports camera corner positioning. This positioning method is difficult to obtain accurate depth information. Considering that the LiDAR (Light Detection And Ranging) sensor can obtain accurate depth information by measuring the time interval between laser emission and reception, the present invention accurately constructs a three-dimensional point cloud model of the environment by adding a LiDAR sensor.

[0049] The initial positioning of the intelligent body is also called re-positioning. The current posture is determined by the historical map. The original initial positioning algorithm obtains the surrounding corner points at the starting position, searches the key frame database for a key frame similar to the current frame as a matching frame through the bag-of-words vector search, establishes data association and calculates the posture of the current frame, thereby achieving positioning. This method requires matching with all key frames, which is not efficient. Considering that the robot is charging at the charging pile position before receiving the work order and the map origin is set at the No. 1 charging pile position, the search range is narrowed with this prior posture. Therefore, the embodiment of the present application designs an efficient strategy to improve the reliability of matching: only key frames within ten meters of the map origin are matched, which greatly improves the matching efficiency. The similarity score calculation method is as shown in formula (1):

[0050]

[0051] Here, s(v1, v2) is the similarity score of the two bag-of-words vectors. v1 and v2 are two bag-of-words vectors. A bag-of-words vector is a statistical histogram vector generated by counting the words appearing in the image relative to the words in the bag-of-words. Then, all candidate frames within a ten-meter radius are traversed, and frames with a common word count greater than the threshold minCommonWords and a word match greater than minScore are selected and stored in ScoreAndMatch. The frames in ScoreAndMatch are sorted by similarity score, and the one with the highest score is selected as the candidate pose for the current agent.

[0052] During this process, the intelligent agents and the central computer exchange positioning information through wireless communication modules to ensure the consistency and synchronization of all intelligent agents in the global coordinate system.

[0053] S2: Semantic segmentation image.

[0054] Use deep learning models to perform semantic segmentation on images captured by the camera and extract semantic information of different objects and scene elements.

[0055] More specifically: After acquiring the image captured by the camera, the present invention uses the pre-trained BiSeNetV2 deep learning model to perform real-time pixel-level semantic segmentation of the image, such as pedestrians, vehicles, buildings, etc. Through this process, each image is converted into a pixel-level label map, in which each pixel is assigned a semantic category. In addition, in order to improve the accuracy of semantic segmentation of domestic park data, the embodiment of the present application collected part of the park data for end-to-end training of the model, and used data enhancement technology to improve the robustness of the model to different lighting and weather conditions. The hardware and software configuration of the semantic segmentation training system is shown in Table 1.

[0056] Table 1. Software and hardware configuration of semantic segmentation training system

[0057]

[0058] The segmentation accuracy data of different rounds is as follows Figure 4 As shown in the figure, the segmentation accuracy stabilizes at around 87.247% after 100 rounds of training, and each frame takes about 8.3 milliseconds, meeting the requirements of real-time semantic segmentation.

[0059] S3: Outlier removal.

[0060] By setting the point cloud processing algorithm, outliers in the lidar point cloud can be identified and removed.

[0061] More specifically: In the lidar point cloud data, abnormal points may be generated due to the influence of various factors (such as reflection, occlusion, etc.).

[0062] (1) Data preprocessing: Considering the physical characteristics of certain points of the lidar (point cloud spot size, signal-to-noise ratio), the “good points” are selected through the point ranking mechanism. Figure 5 Select the schematic diagram for good points.

[0063] ① Distance: remove points that are too close or too far. X, Y, and Z represent the front, left, and top direction coordinate axes of the radar respectively. The three-dimensional coordinate point P output by the radar can be expressed as (P x ,P y ,P z ) indicates that the distance calculation formula from the measuring point P to the radar is:

[0064]

[0065] Where D(P) is the distance from point P to the radar, x, y, and z are the x, y, and z axis coordinates of point P;

[0066] ② Deflection angle: For example, at point f, due to the irregular scanning characteristics of solid-state laser radar, the trajectory curvature of the radar's FOV edge scanning points is large, which will affect subsequent feature extraction. Therefore, the FOV edge points must be eliminated. Based on the point's offset angle, that is, the angle between the radar ray and the x-axis, points close to the edge are eliminated. The point offset angle calculation formula is:

[0067]

[0068] Where, is the offset angle of point P;

[0069] The size of the deflection angle can be used to determine whether it is close to the edge. For LIVOX MID-70, if the deflection angle is greater than 35°, it will be discarded.

[0070] ③Incident angle: For example, at point b, when the incident angle is close to 0° or 180°, the laser spot becomes larger, and the laser beam ranging range becomes the average distance of all ranges within the spot, rather than the distance of a specific point, resulting in inaccurate ranging accuracy. Therefore, points with incident angles close to 0° or 180° should be eliminated. Let P b The adjacent point is P a 、P c , then point P b The angle of incidence is calculated as:

[0071]

[0072] Where, θ(P b ) is point P b The incident angle, P a 、P c Point P b adjacent points.

[0073] The embodiments of the present application identify and remove these outliers through the above steps. In addition, in some embodiments, step S3 also includes analyzing the spatial distribution of outliers to identify and remove systematic errors caused by environmental features (such as glass surfaces). Through this process, the quality of the LiDAR point cloud data is significantly improved, providing reliable basic data for subsequent depth interpolation and 3D reconstruction.

[0074] S4: LiDAR points projected onto the semantically segmented image.

[0075] Leveraging the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected onto the corresponding locations in the semantic segmentation image, ensuring the spatial correspondence between the point cloud data and the image data.

[0076] More specific:

[0077] The radar and camera of each intelligent agent need to be accurately calibrated. The lidar used in this embodiment is the RS-lidar-32 from RoboSense, and the camera is a monocular camera, model DH-DAE-C3210F. Due to installation errors and all errors in the camera itself, the lidar and camera on each intelligent agent need to be calibrated. The specific calibration process of an intelligent agent camera and lidar is as follows:

[0078] S41: Start the camera and lidar on the agent.

[0079] S42: Place the calibration plate in the working area of ​​the intelligent agent, and ensure that multiple feature points of the calibration plate are within the field of view of the camera and lidar.

[0080] S43: Run the calibration algorithm plycal, which automatically captures feature points on the calibration plate and calculates the relative position and direction between the camera and lidar.

[0081] S44: The plycal algorithm will generate a calibration parameter that includes the camera intrinsic and extrinsic parameters as well as the lidar intrinsic and extrinsic parameters: the calibration parameters are Calibration params (roll, pitch, yaw, X, Y, Z): 0.7854, -0.5236, 1.0472, 0.2345, -0.0987, 0.1592.

[0082] Among them, Roll: the rotation angle of the camera around the x-axis, Pitch: the rotation angle of the camera around the y-axis, Yaw: the rotation angle of the camera around the z-axis, X: the position of the camera relative to the lidar on the x-axis, Y: the position of the camera relative to the lidar on the y-axis, Z: the position of the camera relative to the lidar on the z-axis.

[0083] Using these calibration parameters, the point cloud data is projected into the camera coordinate system corresponding to the semantically segmented image from step S2. This step involves converting the 3D point cloud into 2D pixel coordinates, thereby generating a sparse depth map that matches the resolution of the semantically segmented image. The field of view parameters of the LiDAR and monocular camera used in this embodiment are shown in Tables 2 and 3.

[0084] Table 2 RS-lidar-32 field of view parameters

[0085]

[0086] Table 3 DH-DAE-C3210F field of view parameters

[0087]

[0088] In some embodiments, since the horizontal field of view of the laser radar selected in the embodiment of the present application is 360° and the horizontal field of view of the monocular camera is 106°, it can be seen that the laser radar has a horizontal field of view of about 254° that cannot match the camera. In order to improve the efficiency of point cloud and image projection and simplify the calculation, this patent retains the point cloud between the horizontal field of view angle of the laser radar {-53° to +53°}, and the remaining point clouds are filtered through a bandpass filter. Then, the laser radar point cloud data is projected onto the camera image plane through the perspective projection method. The specific process is as follows: Figure 6 The flowchart for projecting LiDAR point cloud data onto the camera image plane is shown in this figure. This projection operation is performed on a central computer, reducing the computational intensity of the agent and saving the cost of the processor loaded on the agent. Specifically, the steps for projecting LiDAR point cloud data onto the camera image plane include:

[0089] S401: Initialize ROS node;

[0090] S402: The central computer receives the agent image and point cloud data;

[0091] S403: Synchronize image and point cloud data;

[0092] S404: Read the calibration file;

[0093] S405: Converting the point cloud data into a camera coordinate system;

[0094] S406: Publish the position of each point in the image.

[0095] S5: Interpolate to obtain a dense depth image.

[0096] A dense depth image is obtained through upsampling method to provide more detailed depth information for 3D reconstruction.

[0097] More specifically: The embodiment of the present application uses an upsampling algorithm to generate a dense depth image. First, step S4 is performed to obtain a sparse depth map corresponding to the monocular image. Then, the sparse depth map is upsampled using the nearest neighbor interpolation method. Figure 7 .

[0098] The specific steps are as follows:

[0099] S51: Acquisition of semantic labels of point clouds: Obtain semantic labels corresponding to the point clouds obtained in step S4 based on the corresponding positions of the point clouds and images.

[0100] S52: Point cloud upsampling: For each object with a semantic label, the nearest neighbor upsampling algorithm is used to interpolate the sparse depth map, fill in the missing depth information, and generate a dense depth map that matches the resolution of the monocular image. The nearest neighbor upsampling algorithm is based on the principle of nearest neighbors. It finds the nearest neighbor points in the existing point cloud data and generates new points based on their position and attribute information. The formula for nearest neighbor interpolation can be simplified as follows: If point P i is the sampling point to be sampled, P j If it is its nearest neighbor, then the new point P new The position can be calculated as follows:

[0101]

[0102] Formula (5) means inserting a point between the original point and its nearest neighbor. The position of the new point is the average position of the two. This step is repeated until a dense depth map matching the resolution of the monocular image is generated.

[0103] S53: Semantic fusion: During the interpolation process, the label information in the semantic segmentation image is stored in the corresponding point cloud channel to achieve the fusion of semantic information and depth information.

[0104] S6: Multi-agent network data synchronization

[0105] Through the central computer and communication modules, multiple agents are synchronized with the central computer. Each agent regularly sends its sensor data and status information to the central computer, which is responsible for coordinating these data to ensure their consistency in time.

[0106] More specifically, to ensure accurate data synchronization within a multi-agent system, this paper designs a timestamp-based network synchronization mechanism. Each agent, when collecting data, timestamps it and sends it to a central computer via a communication module. The central computer receives data from all agents and sorts and synchronizes it based on the timestamps. The specific steps are as follows:

[0107] S61: Data Collection: Each agent collects sensor data (images, point clouds, IMU data, and surveillance video streams) and timestamps each piece of data. This timestamp is encoded in the header file, as shown in Table 4. The lightweight and efficient structured data storage format Protobuf (Protocol Buffers) is selected. Protobuf extends messages by adding new fields without affecting old applications, making it suitable for upgrades without service interruption.

[0108] Table 4 Sensor data header file example

[0109]

[0110] S62: Data Transmission: The wireless communication module transmits time-stamped data to a central computer. Efficient data compression and transmission algorithms are designed to reduce the size of transmitted data while ensuring data integrity and accuracy. Encryption and anonymity technologies are introduced to ensure the security of transmitted data.

[0111] S63: Data synchronization: The central computer receives data from all intelligent agents, synchronizes and integrates them according to timestamps, and ensures the consistency of data in time.

[0112] S64: Data storage: The synchronized data is stored in a central database, providing a basis for subsequent data fusion and 3D reconstruction.

[0113] S7: Multi-agent collaborative map construction.

[0114] Realize map registration between multiple agents and build a globally consistent 3D map.

[0115] More specifically: In multi-agent systems, data fusion and map registration are key steps to achieve globally consistent maps.

[0116] In some embodiments, registration is performed only on a central computer, rather than on each agent. This greatly reduces the processing pressure on the agent, allowing the agent to use a lower-configuration processor, saving the cost of the agent processor. This patent uses the ICP (Iterative Closest Point) algorithm to perform map registration on the central computer. This registration method has the advantages of low resource consumption, fast registration speed, and high accuracy. Graph optimization is used to optimize the map to eliminate cumulative errors. The specific steps are as follows:

[0117] S71: Initial Registration: The central computer uses the ICP algorithm to initially register each agent's local map with the global map. The ICP algorithm iteratively adjusts the position and pose of each local map to align it with the global map by minimizing the distance between point clouds. Table 5 shows the pseudocode for generating a multi-agent collaborative map using ICP registration.

[0118] Table 5 Pseudo code for ICP registration to generate multi-agent collaborative map

[0119]

[0120]

[0121] S72: Optimize registration: Based on the initially registered global map, use a global optimization algorithm to further optimize the registration results of each local map to ensure the consistency and accuracy of the global map.

[0122] This step takes into account computer performance and uses graph optimization to optimize map registration, mainly reducing computational overhead through subgraph transformation and operator fusion.

[0123] S73: Map update: Integrate the optimized local map into the global map and update the global map in real time.

[0124] S8: Redundancy filtering and 3D reconstruction.

[0125] Filter redundant data to improve the efficiency and accuracy of 3D reconstruction.

[0126] More specifically: In a multi-agent system, the presence of redundant data may affect the efficiency and accuracy of 3D reconstruction. This embodiment of the application uses a threshold and clustering algorithm to filter out redundant data and uses the Poisson Surface Reconstruction algorithm for 3D reconstruction. The specific steps are as follows:

[0127] S81: Cluster filtering: Use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to cluster the point cloud data and remove isolated points and noise points.

[0128] S82: 3D Reconstruction: Use the Poisson Surface Reconstruction algorithm to reconstruct a 3D model from the filtered point cloud data. Poisson Surface Reconstruction reconstructs a smooth 3D surface by solving the Poisson equation. This algorithm uses the normal vector information of the point cloud to generate the 3D surface by constructing an implicit function. The core of this method is to obtain the implicit surface function (p)\phi(p)(p) by solving the Poisson equation, which is:

[0129]

[0130] Where Δ is the Laplace operator, which represents the second-order spatial derivative; (p) is the implicit surface function, which represents the scalar value of each point; F(p) is the normal vector field of each point in the point cloud. The normal vector field F(p) is an important feature of point cloud data, which represents the normal vector of each point. Assume that at each point p i On the left, the normal vector F(p i ) is known (by neighborhood estimation), the source term of the Poisson equation is the divergence of the normal vector field To solve the Poisson equation, it is discretized into a system of linear equations. By gridding the point cloud data, the equation can be expressed as:

[0131] AΦ=b (7)

[0132] Where A is the discretized Laplace matrix, which represents the neighborhood relationship of each point in space; Φ is the discrete value of the implicit surface function (p); and b is the source term consisting of the point cloud normal vector F(p).

[0133] By solving the linear system equation (7), the implicit surface function value Φ at each point can be obtained. Finally, the zero isosurface of the implicit surface function (p) (i.e., (p) = 0) defines the three-dimensional surface:

[0134] S={p│φ(p)=0} (8)

[0135] Where S is a three-dimensional surface and p is a point with a scalar value of 0.

[0136] This zero isosurface represents the 3D reconstructed surface of the point cloud data.

[0137] S83: Model optimization: Optimize the reconstructed 3D model, remove unnecessary details and noise, and ensure the accuracy and quality of the model.

[0138] S9: Network sharing and map real-time update and maintenance.

[0139] The purpose of step S29 is to achieve real-time updating and maintenance of the map, ensuring the timeliness and accuracy of the three-dimensional semantic map.

[0140] More specifically: Through the central computer and communication module, the map is updated and maintained in real time. The central computer regularly receives sensor data from each agent and uses this data to update the global map. The updated map data is sent back to each agent to ensure that they have the latest environmental information. Figure 8 The park inspection significance map depicts the data flow. Figure 8 In the figure, the central computer 801 is connected to the intelligent body 802 by signal, 803 represents the park road, and 804 represents the data flow direction.

[0141] In summary, the method provided in the embodiments of the present application has at least the following beneficial effects:

[0142] 1) Multi-sensor data fusion and precise registration: How to achieve high-precision fusion of different sensor data and how to ensure precise registration between LiDAR point clouds and semantic segmentation images to improve the accuracy and reliability of 3D semantic reconstruction.

[0143] 2) Outlier removal and depth image optimization: An effective algorithm is used to identify and remove outliers in LiDAR data, and robust interpolation techniques are used to optimize depth images to obtain higher quality 3D reconstruction results.

[0144] 3) Multi-agent real-time map update: Design a multi-agent network synchronization mechanism to achieve efficient data sharing and rapid map registration, and develop real-time map update and maintenance technology to dynamically reflect changes in the campus environment and enhance the adaptability and robustness of the system.

[0145] The embodiment of the present application also provides a three-dimensional semantic reconstruction device for multi-agent perception fusion in a park, such as Figure 9 As shown, the device includes:

[0146] The synchronous positioning module 901 is configured for time synchronization and initial positioning: initializing a global coordinate system and a global map, and using the sensor fusion of the lidar and camera in combination with the SLAM algorithm to achieve the initial positioning of the intelligent agent in the global map at the initial position in the charging room;

[0147] The image segmentation module 902 is configured to semantically segment the image: performing semantic segmentation on the image captured by the camera based on a deep learning model to extract semantic information of different objects and scene elements;

[0148] The outlier removal module 903 is configured to remove outliers: identifying and removing outliers in the LiDAR point cloud by setting a point cloud processing algorithm;

[0149] Image projection module 904 is configured to project the LiDAR points onto the semantically segmented image: utilizing the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected onto the corresponding positions of the semantically segmented image to obtain a sparse depth map corresponding to the monocular image;

[0150] The image interpolation module 905 is configured to interpolate and obtain a dense depth image: upsampling the sparse depth image using a nearest neighbor interpolation method;

[0151] The data synchronization module 906 is configured to synchronize data in a multi-agent network: each agent adds a timestamp to the collected data when collecting data, and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp;

[0152] Map construction module 907 is configured to construct a multi-agent collaborative map: the central computer uses an iterative closest point algorithm to initially align the local maps of each agent with the global map. Based on the initially aligned global map, a global optimization algorithm is used to further optimize the alignment results of each local map. The optimized local maps are then integrated into the global map, and the global map is updated in real time.

[0153] The 3D reconstruction module 908 is configured to perform redundancy filtering and 3D reconstruction: performing redundancy filtering by setting a threshold and a clustering algorithm, and performing 3D reconstruction;

[0154] The update and maintenance module 909 is configured for network sharing and real-time map update and maintenance: the central computer regularly receives sensor data from each intelligent agent and uses the sensor data of each intelligent agent to update the global map. The updated map data is sent back to each intelligent agent.

[0155] It should be noted that the device described in this embodiment and the method described previously belong to the same technical concept, have the same technical principles, and can achieve the same beneficial effects, so they will not be described in detail here.

[0156] An embodiment of the present application also provides a three-dimensional semantic reconstruction device for multi-agent perception fusion in a campus, and the system includes: a memory for storing a computer program; and a processor for executing the computer program to implement the method described in any of the above embodiments.

[0157] An embodiment of the present application further provides a non-transitory computer-readable storage medium storing instructions, and when the instructions are executed by a processor, the method described in any of the above embodiments is executed.

[0158] The above implementation modes are only used to illustrate the present application and are not intended to limit the present application. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions also fall within the scope of the present application, and the scope of patent protection of the present application shall be defined by the claims.

Claims

1. A 3D semantic reconstruction method based on multi-agent perception fusion in a park, characterized by: The method comprises: Time synchronization and initial positioning: Initialize a global coordinate system and global map. At the initial position in the charging room, use lidar and camera sensor fusion combined with SLAM algorithms to achieve initial positioning of the intelligent agent in the global map. Semantic segmentation of images: Perform semantic segmentation on camera-captured images based on deep learning models to extract semantic information of different objects and scene elements; Outlier removal: Identify and remove outliers in the LiDAR point cloud by setting a point cloud processing algorithm; LiDAR point projection to the semantically segmented image: Using the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected to the corresponding position of the semantically segmented image to obtain a sparse depth map corresponding to the monocular image; Interpolation to obtain dense depth image: Use the nearest neighbor interpolation method to upsample the sparse depth map; Multi-agent network data synchronization: When each agent collects data, it timestamps the data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp; Constructing a multi-agent collaborative map: The central computer uses an iterative closest point algorithm to initially align the local maps of each agent with the global map. Based on the initially aligned global map, a global optimization algorithm is used to further optimize the alignment results of each local map. The optimized local maps are then integrated into the global map, and the global map is updated in real time. Redundancy filtering and 3D reconstruction: Redundancy filtering is performed by setting thresholds and clustering algorithms, and 3D reconstruction is performed; Network sharing and real-time map update and maintenance: The central computer regularly receives sensor data from each intelligent agent and uses the sensor data of each intelligent agent to update the global map. The updated map data is then sent back to each intelligent agent. Redundancy filtering and 3D reconstruction, including: Use clustering algorithms to cluster point cloud data and remove isolated points and noise points to obtain filtered point cloud data; Based on the filtered point cloud data, the normal vector information of the point cloud is used to generate a three-dimensional surface by constructing an implicit function. The implicit surface function is obtained by solving the Poisson equation, which is: Where Δ is the Laplace operator, which represents the second-order spatial derivative; φ(p) is the implicit surface function, which represents the scalar value of each point; F(p) is the normal vector field of each point in the point cloud; If at each point p i On the left, the normal vector F(p i ) is known, then the source term of the Poisson equation is the divergence of the normal vector field The Poisson equation is discretized into a system of linear equations. By gridding the point cloud data, the Poisson equation can be expressed as: AΦ=b (7) Where A is the discretized Laplace matrix, which represents the neighborhood relationship of each point in space; Φ is the discrete value of the implicit surface function φ(p); b is the source term consisting of the point cloud normal vector F(p); Obtain the implicit surface function value Φ of each point by solving the linear system formula; The three-dimensional surface is defined by the zero isosurface of the implicit surface function φ(p): S={p|φ(p)=0} (8) Where S is a three-dimensional surface and p is a point with a scalar value of 0.

2. The 3D semantic reconstruction method based on multi-agent perception fusion in a park as claimed in claim 1, characterized in that: The time synchronization and initial positioning include: The initial positioning of the agent is performed by the following method: Match keyframes within ten meters of the map origin, and the similarity score between two bag-of-words vectors is calculated as follows: Among them, s(v1,v2) is the similarity score of the two bag-of-words vectors, v1 and v2 are two bag-of-words vectors, and the bag-of-words vector refers to a statistical histogram vector generated by the statistics of the words appearing in the image relative to the words in the bag of words; Traverse all candidate frames within ten meters, select frames with a common word count greater than the threshold minCommonWords and a word matching degree greater than minScore and store them in ScoreAndMatch, sort the frames in ScoreAndMatch by similarity score, and select the one with the highest score as the candidate pose for the current agent.

3. The 3D semantic reconstruction method based on multi-agent perception fusion in a park as claimed in claim 1, characterized in that: The outlier removal includes: The three-dimensional coordinate point P output by the laser radar is converted to (P x ,P y ,P z ) indicates that the distance calculation formula from point P to the radar is: Where D(P) is the distance from point P to the radar, x, y, and z are the x, y, and z axis coordinates of point P; Keep the points whose distance to the laser radar is within the first threshold; According to the offset angle of the point, that is, the angle between the radar ray and the x-axis, the points close to the edge are eliminated. The calculation formula of the offset angle of the point is: Where, is the offset angle of point P; Discard points with deflection angles greater than 35°; Eliminate points whose incident angles are within the second threshold. The calculation formula for the incident angle is: Where, θ(P b ) is point P b The incident angle, P a 、P c Point P b adjacent points.

4. The 3D semantic reconstruction method based on multi-agent perception fusion in a park as claimed in claim 1, characterized in that: LiDAR points are projected onto a semantically segmented image, including: Calibrate the lidar and camera of each agent to obtain calibration parameters; Based on the calibration parameters, the point cloud data is projected into a camera coordinate system corresponding to the semantic segmentation image to generate a sparse depth map that matches the resolution of the semantic segmentation image; Calibrate the lidar and camera of each agent to obtain calibration parameters, including: Start the camera and lidar on the agent; Place the calibration plate within the agent's work area, ensuring that multiple feature points on the calibration plate are within the field of view of the camera and lidar. Run the calibration algorithm plycal to automatically capture feature points on the calibration plate and calculate the relative position and orientation between the camera and lidar. Generate a calibration parameter that includes the camera intrinsic and extrinsic parameters as well as the lidar intrinsic and extrinsic parameters: the calibration parameters are Calibrationparams(Roll,Pitch,Yaw,X,Y,Z):0.7854,-0.5236,1.0472,0.2345,-0.0987,0.1592; Among them, Calibrationparams is the calibration parameter, Roll is the rotation angle of the camera around the x-axis, Pitch is the rotation angle of the camera around the y-axis, Yaw is the rotation angle of the camera around the z-axis, X is the position of the camera relative to the lidar on the x-axis, Y is the position of the camera relative to the lidar on the y-axis, and Z is the position of the camera relative to the lidar on the z-axis; Based on the calibration parameters, projecting the point cloud data into a camera coordinate system corresponding to the semantic segmentation image includes retaining point clouds between a laser radar horizontal field of view angle of -53° and +53°, filtering the remaining point clouds through a bandpass filter, and projecting the laser radar point cloud data onto the camera image plane through a perspective projection method.

5. The 3D semantic reconstruction method based on multi-agent perception fusion in a park as claimed in claim 1, characterized in that: Interpolation obtains dense depth images, including: Based on the corresponding positions of the point cloud and the image, obtaining the semantic labels corresponding to the point cloud; wherein the corresponding positions of the point cloud and the image are determined according to the sparse depth map; For each object with a semantic label, the nearest neighbor upsampling algorithm is used to interpolate the sparse depth map, fill in the missing depth information, and generate a dense depth map that matches the resolution of the monocular image. The nearest neighbor upsampling algorithm is based on the principle of the nearest neighbor, and finds the nearest neighbor points in the existing point cloud data, and generates new points based on their position and attribute information. The formula of the nearest neighbor interpolation is expressed as: If point P i is the sampling point to be sampled, P j It's P i The nearest neighbor point is the new point P new The position is calculated as follows: Formula (5) means inserting a point between the original point and its nearest neighbor. The position of the new point is the average position of the two. Repeat the interpolation step until a dense depth map matching the resolution of the monocular image is generated. During the interpolation process, the label information in the semantic segmentation image is stored in the corresponding point cloud channel to obtain a dense depth image.

6. The 3D semantic reconstruction method based on multi-agent perception fusion in a park as claimed in claim 1, characterized in that: Multi-agent network data synchronization, including: Each agent collects sensor data, adds a timestamp to each piece of data, and encodes the timestamp into the header file; wherein the sensor data includes images, point clouds, IMU data, and monitoring video streams; The data with time stamp is sent to the central computer through the wireless communication module, and encryption and anonymity technology are introduced during the data transmission process; The central computer receives data from all agents, synchronizes and integrates them according to timestamps, and obtains synchronized data; The synchronized data is stored in a central database.

7. A 3D semantic reconstruction device based on multi-agent perception fusion in a park, characterized by: The device comprises: The synchronous positioning module is configured for time synchronization and initial positioning: it initializes a global coordinate system and a global map. At the initial position of the charging room, it uses the sensor fusion of the lidar and camera, combined with the SLAM algorithm, to achieve the initial positioning of the intelligent agent in the global map. The image segmentation module is configured to semantically segment images: it performs semantic segmentation on the images captured by the camera based on a deep learning model to extract semantic information of different objects and scene elements; The outlier removal module is configured to identify and remove outliers in the lidar point cloud by setting a point cloud processing algorithm; The image projection module is configured to project the LiDAR points onto the semantically segmented image: utilizing the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected onto the corresponding positions in the semantically segmented image to obtain a sparse depth map corresponding to the monocular image; The image interpolation module is configured to interpolate and obtain a dense depth image: the sparse depth image is upsampled using the nearest neighbor interpolation method; The data synchronization module is configured to synchronize data in a multi-agent network: when each agent collects data, it timestamps the collected data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp; The map construction module is configured to construct a multi-agent collaborative map: the central computer uses an iterative closest point algorithm to initially align the local maps of each agent with the global map. Based on the initially aligned global map, a global optimization algorithm is used to further optimize the alignment results of each local map. The optimized local maps are then integrated into the global map, and the global map is updated in real time. The 3D reconstruction module is configured for redundancy filtering and 3D reconstruction: redundancy filtering is performed by setting a threshold and a clustering algorithm, and 3D reconstruction is performed; The update and maintenance module is configured for network sharing and real-time map update and maintenance: the central computer regularly receives sensor data from each intelligent agent and uses the sensor data of each intelligent agent to update the global map. The updated map data is then sent back to each intelligent agent. The 3D reconstruction module is further configured to: Use clustering algorithms to cluster point cloud data and remove isolated points and noise points to obtain filtered point cloud data; Based on the filtered point cloud data, the normal vector information of the point cloud is used to generate a three-dimensional surface by constructing an implicit function. The implicit surface function is obtained by solving the Poisson equation, which is: Where Δ is the Laplace operator, which represents the second-order spatial derivative; φ(p) is the implicit surface function, which represents the scalar value of each point; F(p) is the normal vector field of each point in the point cloud; If at each point p i On the left, the normal vector F(p i ) is known, then the source term of the Poisson equation is the divergence of the normal vector field The Poisson equation is discretized into a system of linear equations. By gridding the point cloud data, the Poisson equation can be expressed as: AΦ=b (7) Where A is the discretized Laplace matrix, which represents the neighborhood relationship of each point in space; Φ is the discrete value of the implicit surface function φ(p); b is the source term consisting of the point cloud normal vector F(p); Obtain the implicit surface function value Φ of each point by solving the linear system formula; The three-dimensional surface is defined by the zero isosurface of the implicit surface function φ(p): S={p|φ(p)=0} (8) Where S is a three-dimensional surface and p is a point with a scalar value of 0.

8. A 3D semantic reconstruction device based on multi-agent perception fusion in a park, characterized by: The device comprises: memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 6. 9 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, executes the method according to claim 1 .

Images