Park multi-agent perception fusion three-dimensional semantic reconstruction method, device and equipment and storage medium

Abstract

The invention relates to the technical field of park management and automatic driving, and discloses a park multi-agent perception fusion three-dimensional semantic reconstruction method, device and equipment and a storage medium. The method comprises the steps of time synchronization and initial positioning, semantic segmentation of an image, abnormal point removal, projection of a laser radar point to the image after semantic segmentation, acquisition of a dense depth image through interpolation, multi-agent network data synchronization, construction of a multi-agent collaborative map, redundancy filtering and three-dimensional reconstruction, and network sharing and real-time updating and maintenance of the map. According to the invention, through combination of the laser radar, the camera and the IMU data, the advanced SLAM algorithm and the deep learning technology are utilized, and high-precision multi-sensor data fusion and accurate registration are realized. Therefore, the accuracy of three-dimensional semantic reconstruction is improved, the reliability of the system is enhanced, and the system can stably work under various environmental conditions.

Description

Technical Field

[0001] The present application relates to the field of park management and autonomous driving technology, and in particular to a three-dimensional semantic reconstruction method, device, equipment and storage medium for multi-agent perception fusion in a park. Background Art

[0002] In today's campus management and autonomous driving fields, the collaborative work of multi-agent systems is essential to achieve efficient and accurate 3D semantic reconstruction. However, existing methods face many challenges in achieving this goal. First, traditional multi-agent perception systems often rely on single sensor data, such as lidar or camera, which limits the system's perception range and accuracy. In addition, the accuracy of semantic segmentation and point cloud registration during data fusion directly affects the quality of the final reconstruction result.

[0003] When processing LiDAR point cloud data, existing technologies usually lack effective methods for outlier removal and deep image interpolation, which can lead to errors and discontinuities in the reconstructed 3D models. At the same time, network synchronization and data sharing between multiple agents often have delays and efficiency issues. In addition, existing map registration is often inefficient when processing large-scale data, and it is difficult to achieve real-time updates and maintenance. Summary of the invention

[0004] In order to solve the above problems, the present application provides a three-dimensional semantic reconstruction method, device, equipment and storage medium for multi-agent perception fusion of a park, so as to solve the problems existing in the prior art in multi-agent perception fusion, image and point cloud registration, and three-dimensional semantic reconstruction, and provide an efficient and accurate solution for the fields of park management and autonomous driving.

[0005] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0006] In a first aspect, the present application provides a method for 3D semantic reconstruction of a park multi-agent perception fusion, the method comprising:

[0007] Time synchronization and initial positioning: Initialize a global coordinate system and global map, and use the sensor fusion of lidar and camera in the initial position in the charging room, combined with the SLAM algorithm, to achieve the initial positioning of the intelligent body in the global map;

[0008] Semantic segmentation images: Perform semantic segmentation on the images captured by the camera based on the deep learning model to extract the semantic information of different objects and scene elements;

[0009] Outlier removal: Identify and remove outliers in the LiDAR point cloud by setting the point cloud processing algorithm;

[0010] LiDAR point projection to the semantically segmented image: Using the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected to the corresponding position of the semantically segmented image to obtain a sparse depth map corresponding to the monocular image.

[0011] Interpolation to obtain dense depth image: Use the nearest neighbor interpolation method to upsample the sparse depth map;

[0012] Multi-agent network data synchronization: When each agent collects data, it timestamps the collected data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp;

[0013] Constructing a multi-agent collaborative map: The central computer uses an iterative closest point algorithm to initially align the local maps of each agent with the global map. Based on the initially aligned global map, a global optimization algorithm is used to further optimize the alignment results of each local map, and the optimized local maps are integrated into the global map, and the global map is updated in real time.

[0014] Redundancy filtering and 3D reconstruction: Redundancy filtering is performed by setting thresholds and clustering algorithms, and 3D reconstruction is performed;

[0015] Network sharing and real-time map update and maintenance: The central computer regularly receives sensor data from each intelligent agent and uses the sensor data of each intelligent agent to update the global map. The updated map data is sent back to each intelligent agent.

[0016] In a second aspect, the present application provides a three-dimensional semantic reconstruction device for multi-agent perception fusion in a park, the device comprising:

[0017] The synchronous positioning module is configured for time synchronization and initial positioning: a global coordinate system and a global map are initialized, and the initial position of the agent in the global map is achieved by using the sensor fusion of the lidar and the camera in combination with the SLAM algorithm;

[0018] The image segmentation module is configured to semantically segment the image: semantically segment the image captured by the camera based on the deep learning model to extract the semantic information of different objects and scene elements;

[0019] The outlier removal module is configured to remove outliers: identify and remove outliers in the lidar point cloud by setting a point cloud processing algorithm;

[0020] The image projection module is configured to project the lidar points onto the semantically segmented image: utilizing the geometric relationship between the lidar and the camera, the lidar point cloud data is projected onto the corresponding position of the semantically segmented image to obtain a sparse depth map corresponding to the monocular image;

[0021] The image interpolation module is configured to interpolate and obtain a dense depth image: the sparse depth image is upsampled using the nearest neighbor interpolation method;

[0022] The data synchronization module is configured for multi-agent network data synchronization: when each agent collects data, it timestamps the collected data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp;

[0023] The map construction module is configured to construct a multi-agent collaborative map: the central computer uses an iterative closest point algorithm to initially align the local map of each agent with the global map, and based on the initially aligned global map, uses a global optimization algorithm to further optimize the alignment results of each local map, merges the optimized local map into the global map, and updates the global map in real time;

[0024] The 3D reconstruction module is configured for redundant filtering and 3D reconstruction: redundant filtering is performed by setting a threshold and a clustering algorithm, and 3D reconstruction is performed;

[0025] The update and maintenance module is configured for network sharing and real-time map update and maintenance: the central computer periodically receives sensor data from each intelligent agent, and uses the sensor data of each intelligent agent to update the global map, and the updated map data is sent back to each intelligent agent.

[0026] In a third aspect, the present application provides a three-dimensional semantic reconstruction device for multi-agent perception fusion in a campus, and the system includes: a memory for storing a computer program; and a processor for executing the computer program to implement the method described above.

[0027] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, executes the method described above.

[0028] This application has at least the following beneficial effects:

[0029] 1. Improve fusion accuracy and reliability: This application combines lidar, camera and IMU data, and uses advanced SLAM algorithms and deep learning technology to achieve high-precision multi-sensor data fusion and precise registration. This not only improves the accuracy of 3D semantic reconstruction, but also enhances the reliability of the system, enabling it to work stably under various environmental conditions.

[0030] 2. Optimize data processing and reconstruction quality: Through effective outlier removal algorithms and deep image optimization technology with superimposed semantic information, this application can identify and remove outliers in lidar data, and generate high-quality dense depth images through improved interpolation technology. The use of these technologies significantly improves the quality of 3D reconstruction results and reduces errors.

[0031] 3. Enhance the real-time and adaptability of the system: This application designs a multi-agent network synchronization mechanism and real-time map update and maintenance technology to achieve rapid data sharing and dynamic map updates. This enables the system to respond to environmental changes in a timely manner, improves the real-time and adaptability of park management and autonomous driving, and ensures the safety and efficiency of park inspections. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for three-dimensional semantic reconstruction of multi-agent perception fusion in a park according to an embodiment of the present application is shown.

[0033] Figure 2 A schematic diagram of a charging chamber and a center of a global coordinate system according to an embodiment of the present application is shown.

[0034] Figure 3 A schematic diagram of the intelligent agent construction according to an embodiment of the present application is shown.

[0035] Figure 4 The segmentation accuracy data of different rounds according to the embodiment of the present application are shown.

[0036] Figure 5 A schematic diagram of a good points selection method according to an embodiment of the present application is shown.

[0037] Figure 6 A flowchart of projecting lidar point cloud data onto a camera image plane according to an embodiment of the present application is shown.

[0038] Figure 7 A schematic diagram of laser radar points projected onto an image according to an embodiment of the present application is shown.

[0039] Figure 8 A diagram showing the significance of a campus inspection according to an embodiment of the present application is shown.

[0040] Fig. 9 A structural diagram of a three-dimensional semantic reconstruction device for multi-agent perception fusion in a park according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0042] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0043] In order to solve the problems existing in the prior art in multi-agent perception fusion, image and point cloud registration, and three-dimensional semantic reconstruction, an efficient and accurate solution is provided for the field of park management and autonomous driving. The embodiment of the present application provides a three-dimensional semantic reconstruction method for multi-agent perception fusion in a park, which realizes accurate data synchronization between multiple agents by establishing a global coordinate system and time synchronization. Using semantic segmentation technology, the method can extract accurate semantic information from the image and combine it with the lidar point cloud data to generate high-quality dense depth images through projection and innovative interpolation technology. In addition, the method also includes an efficient outlier removal algorithm to ensure the accuracy of the point cloud data.

[0044] In terms of multi-agent network synchronization, this method proposes an innovative technical means based on network shared data and map registration, which can effectively handle data exchange between multiple agents and achieve fast and accurate map construction. By combining redundant filtering with three-dimensional reconstruction, this method further improves the reconstruction efficiency while ensuring the accuracy and robustness of the reconstruction results. This enables the three-dimensional semantic map to dynamically reflect environmental changes within the park, providing strong support for park management and autonomous driving.

[0045] Specifically, this method can effectively solve the problems of multi-agent network synchronization and map misalignment through sensors installed on the agent (AGV) and the campus network, improve the robustness, adaptability and real-time performance of the system, and ensure the efficient collaborative work of the multi-agent system in a complex dynamic environment. Figure 1 , which is a flow chart of a three-dimensional semantic reconstruction method for multi-agent perception fusion in a park. The three-dimensional semantic reconstruction method for multi-agent perception fusion in a park includes the following steps S1 to S9, which are introduced in detail as follows.

[0046] S1: Time synchronization and initial positioning

[0047] Initialize a global coordinate system and global map through the central computer. Initial position and posture information in the charging room. Use the sensor fusion technology of laser radar and camera, combined with SLAM (Simultaneous Localization and Mapping) algorithm, to achieve the initial positioning of the intelligent body in the global map, such as Figure 2 Schematic diagram of the charging room and the center of the global coordinate system. In the figure, the central computer 202 is set in the charging room 201. A plurality of charging piles 204 are set in the charging room 201. The charging piles 204 are used to charge the intelligent body 203. The structure of the intelligent body 203 is as follows Figure 3 As shown, the intelligent body 203 includes a robot body 307, on which an IMU 301, a laser radar 302, a camera 304 and a network transmission module 306 are provided; wherein 303 is the scanning area of ​​the laser radar 302, and 305 is the viewing angle of the camera.

[0048] More specifically: In an embodiment of the present application, a unified coordinate system is first initialized by the central computer. Each intelligent body (AGV) waits for the central computer to issue an inspection work order at its charging station. After receiving the work order, the intelligent body uses the onboard laser radar and camera to obtain environmental features. This patent uses an improved ORBL-SLAM algorithm for precise positioning. The ORBL-SLAM algorithm is an open source navigation and positioning algorithm based on corner detection and fusion of radar point clouds developed based on the ORB-SLAM algorithm, which can update the position and posture of the intelligent body in real time. ORB-SLAM only supports camera corner positioning. This positioning method is difficult to obtain accurate depth information. Considering that the LiDAR (Light Detection And Ranging) sensor can obtain accurate depth information by measuring the time interval between laser emission and reception, the present invention accurately constructs a three-dimensional point cloud model of the environment by adding a LiDAR sensor.

[0049] The initial positioning of the intelligent body is also called repositioning. The current posture is determined through the historical map. The original initial positioning algorithm obtains the surrounding corner points at the starting position, searches the key frame database for a key frame similar to the current frame as a matching frame through a bag-of-words vector search, establishes data association and calculates the posture of the current frame, thereby achieving positioning. This method requires matching with all key frames, which is not efficient. Considering that the robot is charged at the charging pile position before receiving the work order and the map origin is set at the No. 1 charging pile position, the search range is narrowed with this prior posture. Therefore: This embodiment of the application designs an efficient strategy to improve the reliability of matching: only key frames within ten meters of the map origin are matched, which greatly improves the matching efficiency. The similarity score calculation method is as shown in formula (1):

[0050]

[0051] Among them, s(v1,v2) is the similarity score of the two bag-of-words vectors, v1 and v2 are two bag-of-words vectors, and the bag-of-words vector refers to the statistical histogram vector generated by the statistics of the words appearing in the image relative to the words in the bag-of-words. Then, all candidate frames within ten meters are traversed, and frames with a common number of words greater than the threshold minCommonWords and a word matching degree greater than minScore are selected and stored in ScoreAndMatch. The frames in ScoreAndMatch are sorted by similarity scores, and the one with the highest score is selected as the candidate pose of the current agent.

[0052] During this process, the intelligent agents and the central computer exchange positioning information through wireless communication modules to ensure the consistency and synchronization of all intelligent agents in the global coordinate system.

[0053] S2: Semantic segmentation image.

[0054] Use deep learning models to perform semantic segmentation on images captured by the camera and extract semantic information of different objects and scene elements.

[0055] More specifically: After acquiring the image captured by the camera, the present invention uses a pre-trained BiSeNetV2 deep learning model to perform real-time pixel-level semantic segmentation of the image, such as pedestrians, vehicles, buildings, etc. Through this process, each image is converted into a pixel-level label map, in which each pixel is assigned a semantic category. In addition, in order to improve the accuracy of semantic segmentation of domestic park data, the embodiment of the present application collects part of the park data for end-to-end training of the model, and uses data enhancement technology to improve the robustness of the model to different lighting and weather conditions. The hardware and software configuration of the semantic segmentation training system is shown in Table 1.

[0056] Table 1. Software and hardware configuration of semantic segmentation training system

[0057]

[0058] The segmentation accuracy data of different rounds is as follows: Figure 4 As shown in the figure, the segmentation accuracy stabilizes at around 87.247% after 100 rounds of training, and each frame takes about 8.3 milliseconds, meeting the requirements of real-time semantic segmentation.

[0059] S3: Outlier removal.

[0060] By setting the point cloud processing algorithm, outliers in the LiDAR point cloud can be identified and removed.

[0061] More specifically: In the LiDAR point cloud data, abnormal points may be generated due to the influence of various factors (such as reflection, occlusion, etc.).

[0062] (1) Data preprocessing: Considering the physical characteristics of certain points of the LiDAR (point cloud spot size, signal-to-noise ratio), the “good points” are selected through the point ranking mechanism. Figure 5 Select the diagram for good points.

[0063] ① Distance: remove points that are too close or too far. X, Y, and Z represent the front, left, and top direction coordinate axes of the radar respectively. The three-dimensional coordinate point P output by the radar can be expressed as (P x ,P y ,P z ) indicates that the distance calculation formula from the measuring point P to the radar is:

[0064]

[0065] Where D(P) is the distance from point P to the radar, x, y and z are the x, y and z axis coordinates of point P;

[0066] ② Deflection angle: For example, at point f, due to the irregular scanning characteristics of the solid-state laser radar, the trajectory curvature of the radar's FOV edge scanning point is large, which will affect the subsequent feature extraction, so the FOV edge points should be removed. According to the offset angle of the point, that is, the angle between the radar ray and the x-axis, the points near the edge are removed. The calculation formula for the offset angle of the point is:

[0067]

[0068] In the formula, is the offset angle of point P;

[0069] The size of the deflection angle can be used to determine whether it is close to the edge. For LIVOX MID-70, if the deflection angle is greater than 35°, it will be discarded.

[0070] ③Incident angle: For example, at point b, when the incident angle is close to 0° or 180°, the laser spot becomes larger, and the laser beam ranging range becomes the average distance of all ranges within the spot, rather than the distance of a specific point, resulting in inaccurate ranging accuracy. Therefore, points with incident angles close to 0° or 180° should be eliminated. Assume P b The adjacent point is P a , P c , then point P b The angle of incidence is calculated as:

[0071]

[0072] In the formula, θ(P b ) is point P b The incident angle, P a , P c Point P b adjacent points.

[0073] The embodiment of the present application identifies and removes these outliers through the above steps. In addition, in some embodiments, step S3 also includes analyzing the spatial distribution of outliers to identify and remove systematic errors caused by environmental features (such as glass surfaces). Through this process, the quality of the LiDAR point cloud data is significantly improved, providing reliable basic data for subsequent depth interpolation and three-dimensional reconstruction.

[0074] S4: LiDAR points projected onto the semantically segmented image.

[0075] Using the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected to the corresponding position of the semantic segmentation image, ensuring the spatial correspondence between the point cloud data and the image data.

[0076] More specific:

[0077] It is necessary to accurately calibrate the radar and camera of each intelligent body. The laser radar used in the embodiment of the present application is the RS-lidar-32 of RoboSense, and the camera is a monocular camera, model DH-DAE-C3210F. Due to installation errors and all errors of the camera itself, it is necessary to calibrate the laser radar and camera on each intelligent body. The specific calibration process of an intelligent body camera and laser radar is as follows:

[0078] S41: Start the camera and lidar on the agent.

[0079] S42: Place the calibration plate in the working area of ​​the intelligent body, and ensure that multiple feature points of the calibration plate are within the field of view of the camera and the lidar.

[0080] S43: Run the calibration algorithm plycal, which will automatically capture the feature points on the calibration plate and calculate the relative position and direction between the camera and the lidar.

[0081] S44: The plycal algorithm will generate a calibration parameter containing the camera intrinsic and extrinsic parameters as well as the lidar intrinsic and extrinsic parameters: the calibration parameters are Calibration params (roll, pitch, yaw, X, Y, Z): 0.7854, -0.5236, 1.0472, 0.2345, -0.0987, 0.1592.

[0082] Among them, Roll: the rotation angle of the camera around the x-axis, Pitch: the rotation angle of the camera around the y-axis, Yaw: the rotation angle of the camera around the z-axis, X: the position of the camera relative to the lidar on the x-axis, Y: the position of the camera relative to the lidar on the y-axis, Z: the position of the camera relative to the lidar on the z-axis.

[0083] The point cloud data is projected into the camera coordinate system corresponding to the semantic segmentation image in step S2 through this calibration parameter. This step involves converting the three-dimensional point cloud into two-dimensional pixel coordinates, thereby generating a sparse depth map that matches the resolution of the semantic segmentation image. The field of view parameters of the laser radar and monocular camera selected in the embodiment of the present application are shown in Tables 2 and 3.

[0084] Table 2 RS-lidar-32 field of view parameters

[0085]

[0086] Table 3 DH-DAE-C3210F field of view parameters

[0087]

[0088] In some embodiments, since the horizontal field of view of the laser radar selected in the embodiment of the present application is 360° and the horizontal field of view of the monocular camera is 106°, it can be seen that the laser radar has a horizontal field of view of about 254° that cannot match the camera. In order to improve the efficiency of point cloud and image projection and simplify the calculation, this patent retains the point cloud between the horizontal field of view angle of the laser radar {-53° to +53°}, and the remaining point clouds are filtered through a bandpass filter. Then, the laser radar point cloud data is projected onto the camera image plane through a perspective projection method. The specific process is as follows: Figure 6 As shown in the flowchart of projecting the laser radar point cloud data onto the camera image plane, this projection operation is performed on the central computer, which reduces the computing intensity of the intelligent agent and saves the processor cost loaded on the intelligent agent. Specifically, the steps of projecting the laser radar point cloud data onto the camera image plane include:

[0089] S401: Initialize ROS node;

[0090] S402: The central computer receives the intelligent body image and point cloud data;

[0091] S403: Synchronizing image and point cloud data;

[0092] S404: Read the calibration file;

[0093] S405: converting the point cloud data into a camera coordinate system;

[0094] S406: Publish the position of each point in the image.

[0095] S5: Interpolate to obtain a dense depth image.

[0096] A dense depth image is obtained through upsampling method to provide more detailed depth information for 3D reconstruction.

[0097] More specifically: The embodiment of the present application uses an upsampling algorithm to generate a dense depth image. First, step S4 is performed to obtain a sparse depth map corresponding to the monocular image. Then, the sparse depth map is upsampled using the nearest neighbor interpolation method. Figure 7 .

[0098] The specific steps are as follows:

[0099] S51: Acquisition of semantic labels of point cloud: Obtain the semantic labels corresponding to the point cloud through the corresponding positions of the point cloud and the image obtained in step S4.

[0100] S52: Point cloud upsampling: For each object with a semantic label, the nearest neighbor upsampling algorithm is used to interpolate the sparse depth map, fill in the missing depth information, and generate a dense depth map that matches the resolution of the monocular image. The nearest neighbor upsampling algorithm is based on the principle of the nearest neighbor. It finds the nearest neighbor points in the existing point cloud data and generates new points based on their position and attribute information. The formula for the nearest neighbor interpolation can be simplified as follows: If point P i is the sampling point to be sampled, P j If it is its nearest neighbor, then the new point P new The position can be calculated as follows:

[0101]

[0102] Formula (5) means inserting a point between the original point and its nearest neighbor. The position of the new point is the average position of the two. This step is repeated until a dense depth map matching the resolution of the monocular image is generated.

[0103] S53: Semantic fusion: During the interpolation process, the label information in the semantic segmentation image is stored in the corresponding point cloud channel to achieve the fusion of semantic information and depth information.

[0104] S6: Multi-agent network data synchronization

[0105] Through the central computer and communication modules, the network synchronization of multiple agents and the central computer is realized. Each agent regularly sends its sensor data and status information to the central computer, which is responsible for coordinating these data to ensure their consistency in time.

[0106] More specifically: In order to ensure accurate synchronization of data in a multi-agent system, the present invention designs a network synchronization mechanism based on timestamps. When each agent collects data, it will timestamp the data and send the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamps. The specific steps are as follows:

[0107] S61: Data collection: Each agent collects sensor data (images, point clouds, IMU data, and surveillance video streams) and timestamps each piece of data. This timestamp is encoded in the header file. The schematic content of the header file is shown in Table 4. The lightweight and efficient structured data storage format Protobuf (Protocol Buffers) is selected. Protobuf extends messages by adding new fields without affecting old applications, which makes it possible to upgrade without interrupting services.

[0108] Table 4 Sensor data header file example

[0109]

[0110] S62: Data transmission: The data with time stamp is sent to the central computer through the wireless communication module. An efficient data compression and transmission algorithm is designed to reduce the size of the transmitted data while ensuring the integrity and accuracy of the data. Encryption and anonymity technology are introduced to ensure the security of the transmitted data.

[0111] S63: Data synchronization: The central computer receives data from all agents, synchronizes and integrates them according to timestamps, and ensures the consistency of data in time.

[0112] S64: Data storage: The synchronized data is stored in the central database, providing a basis for subsequent data fusion and 3D reconstruction.

[0113] S7: Multi-agent collaborative map construction.

[0114] Realize map registration between multiple agents and build a globally consistent 3D map.

[0115] More specifically: In multi-agent systems, data fusion and map registration are key steps to achieve a globally consistent map.

[0116] In some embodiments, registration is performed only on the central computer, not on each agent, which greatly reduces the processing pressure of the agent. A processor with a lower configuration can be selected on the agent, saving the cost of the agent processor. This patent uses the ICP (iterative closest point) algorithm to perform map registration on the central computer. This registration method has the advantages of occupying less resources, fast registration speed and high accuracy. Use graph optimization to optimize the map and eliminate cumulative errors. The specific steps are as follows:

[0117] S71: Initial alignment: The central computer uses the ICP algorithm to initially align the local maps of each agent with the global map. The ICP algorithm iteratively adjusts the position and posture of each local map to align it with the global map by minimizing the distance between point clouds. Table 5 is the pseudo code for generating a multi-agent collaborative map by ICP alignment.

[0118] Table 5 Pseudo code for ICP registration to generate multi-agent collaborative map

[0119]

[0120]

[0121] S72: Optimize registration: Based on the initially registered global map, use a global optimization algorithm to further optimize the registration results of each local map to ensure the consistency and accuracy of the global map.

[0122] Considering the computer performance, this step uses graph optimization to optimize the map registration, mainly through sub-graph transformation and operator fusion to reduce the computational overhead.

[0123] S73: Map update: The optimized local map is integrated into the global map, and the global map is updated in real time.

[0124] S8: Redundancy filtering and 3D reconstruction.

[0125] Filter redundant data to improve the efficiency and accuracy of 3D reconstruction.

[0126] More specifically: In a multi-agent system, the existence of redundant data may affect the efficiency and accuracy of 3D reconstruction. The embodiment of the present application performs redundant filtering by setting a threshold and a clustering algorithm, and uses the Poisson Surface Reconstruction algorithm for 3D reconstruction. The specific steps are as follows:

[0127] S81: Cluster filtering: Use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to cluster the point cloud data and remove isolated points and noise points.

[0128] S82: 3D Reconstruction: Use the Poisson Surface Reconstruction algorithm to reconstruct a 3D model from the filtered point cloud data. Poisson Surface Reconstruction reconstructs a smooth 3D surface by solving the Poisson equation. This algorithm uses the normal vector information of the point cloud to generate a 3D surface by constructing an implicit function. The core of this method is to obtain the implicit surface function (p)\phi(p)(p) by solving the Poisson equation, which is:

[0129]

[0130] In the formula, Δ is the Laplace operator, which represents the second-order spatial derivative; (p) is the implicit surface function, which represents the scalar value of each point; F(p) is the normal vector field of each point in the point cloud. The normal vector field F(p) is an important feature of point cloud data, which represents the normal vector of each point. Assume that at each point p i On the left, the normal vector F(p i ) is known (by neighborhood estimation), the source term of the Poisson equation is the divergence of the normal vector field To solve the Poisson equation, it is discretized into a system of linear equations. By gridding the point cloud data, the equation can be expressed as:

[0131] AΦ=b (7)

[0132] Where A is the discretized Laplace matrix, which represents the neighborhood relationship of each point in space; Φ is the discrete value of the implicit surface function (p); and b is the source term consisting of the point cloud normal vector F(p).

[0133] By solving the linear system formula (7), the implicit surface function value Φ at each point can be obtained. Finally, the zero isosurface of the implicit surface function (p) (i.e. (p) = 0) defines a three-dimensional surface:

[0134] S={p│φ(p)=0} (8)

[0135] Where S is a three-dimensional surface and p is a point with a scalar value of 0.

[0136] This zero isosurface represents the 3D reconstructed surface of the point cloud data.

[0137] S83: Model optimization: Optimize the reconstructed 3D model, remove unnecessary details and noise, and ensure the accuracy and quality of the model.

[0138] S9: Network sharing and map real-time update and maintenance.

[0139] The purpose of step S29 is to achieve real-time updating and maintenance of the map to ensure the timeliness and accuracy of the three-dimensional semantic map.

[0140] More specifically: Through the central computer and communication module, the map is updated and maintained in real time. The central computer regularly receives sensor data from each agent and uses this data to update the global map. The updated map data is sent back to each agent to ensure that they have the latest environmental information. Figure 8 The park inspection significance map depicts the flow of data. Figure 8 In the figure, the central computer 801 is connected with the intelligent body 802 by signal, 803 represents the park road, and 804 represents the data flow direction.

[0141] In summary, the method provided in the embodiments of the present application has at least the following beneficial effects:

[0142] 1) Multi-sensor data fusion and precise registration: How to achieve high-precision fusion of different sensor data and how to ensure precise registration between LiDAR point clouds and semantic segmentation images to improve the accuracy and reliability of 3D semantic reconstruction.

[0143] 2) Outlier removal and depth image optimization: An effective algorithm is used to identify and remove outliers in LiDAR data, and a robust interpolation technique is used to optimize the depth image to obtain higher quality 3D reconstruction results.

[0144] 3) Multi-agent real-time map update: Design a multi-agent network synchronization mechanism to achieve efficient data sharing and rapid map registration, and develop real-time map update and maintenance technology to dynamically reflect changes in the park environment and enhance the adaptability and robustness of the system.

[0145] The present application also provides a 3D semantic reconstruction device for multi-agent perception fusion in a park, such as Fig. 9 As shown, the device comprises:

[0146] The synchronous positioning module 901 is configured for time synchronization and initial positioning: initializing a global coordinate system and a global map, at the initial position in the charging room, using the sensor fusion of the laser radar and the camera, combined with the SLAM algorithm, to achieve the initial positioning of the intelligent agent in the global map;

[0147] The image segmentation module 902 is configured to semantically segment the image: semantically segment the image captured by the camera based on the deep learning model to extract semantic information of different objects and scene elements;

[0148] The outlier removal module 903 is configured to remove outliers: identify and remove outliers in the laser radar point cloud by setting a point cloud processing algorithm;

[0149] The image projection module 904 is configured to project the LiDAR points onto the semantically segmented image: utilizing the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected onto the corresponding position of the semantically segmented image to obtain a sparse depth map corresponding to the monocular image;

[0150] The image interpolation module 905 is configured to interpolate and obtain a dense depth image: up-sample the sparse depth image using a nearest neighbor interpolation method;

[0151] The data synchronization module 906 is configured for multi-agent network data synchronization: when each agent collects data, it timestamps the collected data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp;

[0152] The map construction module 907 is configured to construct a multi-agent collaborative map: the central computer uses an iterative closest point algorithm to initially align the local map of each agent with the global map, and based on the initially aligned global map, uses a global optimization algorithm to further optimize the alignment results of each local map, merges the optimized local map into the global map, and updates the global map in real time;

[0153] The 3D reconstruction module 908 is configured for redundancy filtering and 3D reconstruction: performing redundancy filtering by setting a threshold and a clustering algorithm, and performing 3D reconstruction;

[0154] The update and maintenance module 909 is configured for network sharing and real-time map update and maintenance: the central computer periodically receives sensor data from each intelligent agent, and uses the sensor data of each intelligent agent to update the global map, and the updated map data is sent back to each intelligent agent.

[0155] It should be noted that the device described in this embodiment and the method described previously belong to the same technical concept, have the same technical principles, and can achieve the same beneficial effects, so they will not be repeated here.

[0156] An embodiment of the present application also provides a three-dimensional semantic reconstruction device for multi-agent perception fusion in a park, and the system includes: a memory for storing a computer program; and a processor for executing the computer program to implement the method described in any of the above embodiments.

[0157] An embodiment of the present application further provides a non-temporary computer-readable storage medium storing instructions, and when the instructions are executed by a processor, the method described in any of the above embodiments is executed.

[0158] The above implementation modes are only used to illustrate the present application, and are not intended to limit the present application. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions also belong to the scope of the present application, and the scope of patent protection of the present application shall be limited by the claims.

Claims

1. A 3D semantic reconstruction method based on multi-agent perception fusion in a park, characterized in that: The method comprises: Time synchronization and initial positioning: Initialize a global coordinate system and global map, and use the sensor fusion of lidar and camera in the initial position in the charging room, combined with the SLAM algorithm, to achieve the initial positioning of the intelligent body in the global map; Semantic segmentation images: Perform semantic segmentation on the images captured by the camera based on the deep learning model to extract the semantic information of different objects and scene elements; Outlier removal: Identify and remove outliers in the LiDAR point cloud by setting the point cloud processing algorithm; LiDAR point projection to the semantically segmented image: Using the geometric relationship between the LiDAR and the camera, the LiDAR point cloud data is projected to the corresponding position of the semantically segmented image to obtain a sparse depth map corresponding to the monocular image. Interpolation to obtain dense depth image: Use the nearest neighbor interpolation method to upsample the sparse depth map; Multi-agent network data synchronization: When each agent collects data, it timestamps the collected data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp; Constructing a multi-agent collaborative map: The central computer uses an iterative closest point algorithm to initially align the local maps of each agent with the global map. Based on the initially aligned global map, a global optimization algorithm is used to further optimize the alignment results of each local map, and the optimized local maps are integrated into the global map, and the global map is updated in real time. Redundancy filtering and 3D reconstruction: Redundancy filtering is performed by setting thresholds and clustering algorithms, and 3D reconstruction is performed; Network sharing and real-time map update and maintenance: The central computer regularly receives sensor data from each intelligent agent and uses the sensor data of each intelligent agent to update the global map. The updated map data is sent back to each intelligent agent.

2. The 3D semantic reconstruction method of multi-agent perception fusion in a park as claimed in claim 1, characterized in that: The time synchronization and initial positioning include: The initial positioning of the agent is performed by the following method: Match the key frames within ten meters of the map origin, and the calculation formula for the similarity score of two bag-of-words vectors is: Among them, s(v1,v2) is the similarity score of two bag-of-words vectors, v1 and v2 are two bag-of-words vectors, and the bag-of-words vector refers to a statistical histogram vector generated by the statistics of the words appearing in the image relative to the words in the bag of words; Traverse all candidate frames within ten meters, select frames with a common word count greater than the threshold minCommonWords and a word match greater than minScore and store them in ScoreAndMatch, sort the frames in ScoreAndMatch by similarity scores, and select the one with the highest score as the candidate pose for the current agent.

3. The 3D semantic reconstruction method of multi-agent perception fusion in a park as claimed in claim 1, characterized in that: The outlier removal includes: The three-dimensional coordinate point P output by the laser radar is (P x ,P y ,P z ) indicates that the distance calculation formula from point P to the radar is: Where D(P) is the distance from point P to the radar, x, y and z are the x, y and z axis coordinates of point P; Keep the points whose distance to the laser radar is within the first threshold; According to the offset angle of the point, that is, the angle between the radar ray and the x-axis, the points near the edge are eliminated. The calculation formula of the offset angle of the point is: In the formula, is the offset angle of point P; Discard points with deflection angles greater than 35°; The points whose incident angle is within the second threshold are eliminated. The calculation formula of the incident angle is: In the formula, θ(P b ) is point P b The incident angle, P a , P c Point P b adjacent points.

4. The 3D semantic reconstruction method of multi-agent perception fusion in a park as claimed in claim 1, characterized in that: LiDAR points are projected onto a semantically segmented image, including: Calibrate the lidar and camera of each agent to obtain calibration parameters; Based on the calibration parameters, the point cloud data is projected into a camera coordinate system corresponding to the semantic segmentation image to generate a sparse depth map that matches the resolution of the semantic segmentation image; Calibrate the lidar and camera of each agent to obtain calibration parameters, including: Start the camera and lidar on the agent; Place the calibration plate in the working area of ​​the agent, ensuring that multiple feature points of the calibration plate are within the field of view of the camera and lidar; Run the calibration algorithm plycal to automatically capture feature points on the calibration plate and calculate the relative position and direction between the camera and lidar; Generate a calibration parameter containing the camera internal and external parameters as well as the lidar internal and external parameters: the calibration parameters are Calibration params (roll, pitch, yaw, X, Y, Z): 0.7854, -0.5236, 1.0472, 0.2345, -0.0987, 0.1592; Among them, Calibration params are calibration parameters, Roll is the rotation angle of the camera around the x-axis, Pitch is the rotation angle of the camera around the y-axis, Yaw is the rotation angle of the camera around the z-axis, X is the position of the camera relative to the lidar on the x-axis, Y is the position of the camera relative to the lidar on the y-axis, and Z is the position of the camera relative to the lidar on the z-axis; Based on the calibration parameters, projecting the point cloud data into a camera coordinate system corresponding to the semantic segmentation image includes: retaining point clouds between the laser radar horizontal field of view angle of -53° and +53°, filtering the remaining point clouds through a bandpass filter, and projecting the laser radar point cloud data onto the camera image plane through a perspective projection method.

5. The 3D semantic reconstruction method of multi-agent perception fusion in a park as claimed in claim 1, characterized in that: Interpolation to obtain a dense depth image, including: Based on the corresponding positions of the point cloud and the image, a semantic label corresponding to the point cloud is obtained; wherein the corresponding positions of the point cloud and the image are determined according to the sparse depth map; For each object with a semantic label, the nearest neighbor upsampling algorithm is used to interpolate the sparse depth map, fill in the missing depth information, and generate a dense depth map that matches the resolution of the monocular image. The nearest neighbor upsampling algorithm is based on the principle of the nearest neighbor, finds the nearest neighbor point in the existing point cloud data, and generates new points based on its position and attribute information. The formula for the nearest neighbor interpolation is expressed as: If point P i is the sampling point to be sampled, P j YesP i The nearest neighbor point is the new point P new The position is calculated as follows: Formula (5) means inserting a point between the original point and its nearest neighbor. The position of the new point is the average position of the two. Repeat the interpolation steps until a dense depth map matching the resolution of the monocular image is generated. During the interpolation process, the label information in the semantic segmentation image is stored in the corresponding point cloud channel to obtain a dense depth image.

6. The 3D semantic reconstruction method of multi-agent perception fusion in a park as claimed in claim 1, characterized in that: Multi-agent network data synchronization, including: Each agent collects sensor data, adds a timestamp to each piece of data, and encodes the timestamp into a header file; wherein the sensor data includes images, point clouds, IMU data, and surveillance video streams; The data with time stamp is sent to the central computer through the wireless communication module, and encryption and anonymity technology are introduced in the data transmission process; The central computer receives the data from all agents, synchronizes and fuses them according to the timestamps, and obtains synchronized data; The synchronized data is stored in a central database.

7. The 3D semantic reconstruction method of multi-agent perception fusion in a park as claimed in claim 1, characterized in that: Redundancy filtering and 3D reconstruction, including: Use clustering algorithm to cluster point cloud data and remove isolated points and noise points to obtain filtered point cloud data; Based on the filtered point cloud data, the normal vector information of the point cloud is used to generate a three-dimensional surface by constructing an implicit function; wherein the implicit surface function is obtained by solving the Poisson equation, and the Poisson equation is: Where Δ is the Laplace operator, which represents the second-order spatial derivative; φ(p) is the implicit surface function, which represents the scalar value of each point; F(p) is the normal vector field of each point in the point cloud; If at every point p i On the left, the normal vector F(p i ) is known, then the source term of the Poisson equation is the divergence of the normal vector field The Poisson equation is discretized into a system of linear equations. By gridding the point cloud data, the Poisson equation can be expressed as: AΦ=b (7) Where A is the discretized Laplace matrix, which represents the neighborhood relationship of each point in space; Φ is the discrete value of the implicit surface function φ(p); b is the source term composed of the point cloud normal vector F(p); Obtain the implicit surface function value Φ of each point by solving the linear system formula; The three-dimensional surface is defined by the zero isosurface of the implicit surface function φ(p): s={p│φ(p)=0} (8) Where S is a three-dimensional surface and p is a point with a scalar value of 0.

8. A 3D semantic reconstruction device based on multi-agent perception fusion in a park, characterized in that: The device comprises: The synchronous positioning module is configured for time synchronization and initial positioning: a global coordinate system and a global map are initialized, and the initial position of the agent in the global map is achieved by using the sensor fusion of the lidar and the camera in combination with the SLAM algorithm; The image segmentation module is configured to semantically segment the image: semantically segment the image captured by the camera based on the deep learning model to extract the semantic information of different objects and scene elements; The outlier removal module is configured to remove outliers: identify and remove outliers in the lidar point cloud by setting a point cloud processing algorithm; The image projection module is configured to project the lidar points onto the semantically segmented image: utilizing the geometric relationship between the lidar and the camera, the lidar point cloud data is projected onto the corresponding position of the semantically segmented image to obtain a sparse depth map corresponding to the monocular image; The image interpolation module is configured to interpolate and obtain a dense depth image: the sparse depth image is upsampled using the nearest neighbor interpolation method; The data synchronization module is configured for multi-agent network data synchronization: when each agent collects data, it timestamps the collected data and sends the data to the central computer through the communication module. The central computer receives the data of all agents and sorts and synchronizes them according to the timestamp; The map construction module is configured to construct a multi-agent collaborative map: the central computer uses an iterative closest point algorithm to initially align the local map of each agent with the global map, and based on the initially aligned global map, uses a global optimization algorithm to further optimize the alignment results of each local map, merges the optimized local map into the global map, and updates the global map in real time; The 3D reconstruction module is configured for redundant filtering and 3D reconstruction: redundant filtering is performed by setting a threshold and a clustering algorithm, and 3D reconstruction is performed; The update and maintenance module is configured for network sharing and real-time map update and maintenance: the central computer periodically receives sensor data from each intelligent agent, and uses the sensor data of each intelligent agent to update the global map, and the updated map data is sent back to each intelligent agent.

9. A 3D semantic reconstruction device based on multi-agent perception fusion in a park, characterized by: The system comprises: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 7. 10 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, perform the method according to claim 1 .

Images