Three-dimensional dynamic live-action mapping optimization method and device, equipment and storage medium

Abstract

The invention discloses a three-dimensional dynamic live-action mapping optimization method, device and equipment and a storage medium, and the method comprises the steps: starting all bionic quadruped robots, obtaining the inspection data of all bionic quadruped robots, and generating a global map of a current scene according to the inspection data; predicting a future motion path of a dynamic obstacle through an LSTM model according to the global map; determining a planned inspection path according to the future motion path, and performing real-time dynamic three-dimensional mapping according to the planned inspection path in combination with real-time video data to obtain a real-scene mapping optimization result after virtual-real synchronization; according to the method, the equipment state and the environment change can be reflected in time, the timeliness and the safety of inspection decision are improved, the rendering delay is reduced, the positioning precision in a complex environment is improved, the field dynamic change can be accurately mapped, and the speed and the efficiency of three-dimensional dynamic live-action mapping optimization are improved.

Description

Technical Field

[0001] The present invention relates to the field of patrol inspection technology, and in particular to a three-dimensional dynamic real scene mapping optimization method, device, equipment and storage medium. Background Art

[0002] Traditional coal-fired power plant inspection systems rely heavily on static modeling, which is unable to reflect equipment status and environmental changes in real time. Traditional Simultaneous Localization and Mapping (SLAM) solutions, such as Google Cartographer and ORB-SLAM2, have demonstrated high real-time mapping and positioning capabilities in stand-alone environments. However, when applied to multi-robot collaboration, they often lack a unified global optimization strategy and struggle to cope with dynamic obstacles and large-scale scenario expansion.

[0003] Existing virtual-reality fusion solutions include: digital twin platforms use OpenGL / WebGL-based rendering technology to fuse pre-collected video data with three-dimensional models to form a virtual inspection perspective; however, these solutions often rely on offline data or pre-generated models and cannot achieve high-fidelity fusion of real-time on-site data; traditional digital twin platforms are often unable to achieve 1:1 real-scene restoration due to offline data and low-fidelity rendering.

[0004] Most existing technologies rely on offline data collection and static modeling, resulting in slow map data updates. This limitation makes it impossible for the digital twin platform to promptly reflect equipment status and environmental changes, affecting the timeliness and safety of inspection decisions. Existing virtual-reality fusion solutions mainly rely on traditional OpenGL / WebGL rendering technology and lack real-time ray tracing and edge AI processing support. This leads to rendering delays and insufficient accuracy when dynamic video streams are superimposed on three-dimensional models, making it difficult to accurately link the "virtual inspector's" perspective with the actual equipment status. Summary of the Invention

[0005] The main purpose of the present invention is to provide a three-dimensional dynamic real-scene mapping optimization method, device, equipment and storage medium, aiming to solve the technical problems in the existing technology that the digital twin platform cannot timely reflect the equipment status and environmental changes, the timeliness and security of inspection decisions are poor, and there are rendering delays and insufficient accuracy.

[0006] In a first aspect, the present invention provides a three-dimensional dynamic real scene mapping optimization method, the three-dimensional dynamic real scene mapping optimization method comprising the following steps:

[0007] Starting all bionic quadruped robots, obtaining inspection data of all bionic quadruped robots, and generating a global map of the current scene based on the inspection data;

[0008] Predicting the future movement path of the dynamic obstacle through the LSTM model based on the global map;

[0009] A planned inspection path is determined according to the future motion path, and real-time dynamic three-dimensional mapping is performed based on the planned inspection path in combination with real-time video data to obtain a real-scene mapping optimization result after virtual-real synchronization.

[0010] Optionally, the starting all bionic quadruped robots, obtaining inspection data of all bionic quadruped robots, and generating a global map of the current scene according to the inspection data includes:

[0011] Starting all bionic quadruped robots, and aligning clocks of all sensors in all bionic quadruped robots using the Precision Time Protocol (PTP);

[0012] Acquire the laser radar point cloud data and inertial measurement unit (IMU) pose data of each bionic quadruped robot after clock alignment, combine the laser radar point cloud data and the IMU pose data to construct a local submap;

[0013] Obtain visible light video data and thermal imaging data of each bionic quadruped robot after clock alignment, perform splicing correction on the local submaps according to the visible light video data and thermal imaging data, and generate a global map of the current scene.

[0014] Optionally, starting all bionic quadruped robots and aligning clocks of all sensors in all bionic quadruped robots using the Precision Time Protocol (PTP) includes:

[0015] Starting all bionic quadruped robots, and aligning the data timestamps of the lidar, inertial measurement unit (IMU), and dual-spectral camera in all bionic quadruped robots through the Precision Time Protocol (PTP) for synchronization;

[0016] The spatial coordinates of each sensor are calibrated offline using a checkerboard calibration method to align the 3D point cloud of the lidar with the image captured by the dual-spectrum camera in space.

[0017] Optionally, acquiring the laser radar point cloud data and inertial measurement unit (IMU) pose data of each bionic quadruped robot after clock alignment, combining the laser radar point cloud data and the IMU pose data to construct a local submap, includes:

[0018] Scanning the ground features of the current scene using the laser radars of all the bionic quadruped robots to obtain laser radar point cloud data;

[0019] The inertial measurement unit (IMU) of each of the bionic quadruped robots measures the robot's motion speed and angle in real time, and generates IMU pose data based on the motion speed and angle.

[0020] The laser radar point cloud data and the IMU pose data of each bionic quadruped robot are combined to construct a local submap corresponding to each bionic quadruped robot.

[0021] Optionally, acquiring the visible light video data and thermal imaging data of each bionic quadruped robot after clock alignment, performing splicing correction on the local submaps according to the visible light video data and thermal imaging data to generate a global map of the current scene includes:

[0022] Obtain visible light video data and thermal imaging data through the dual-spectral imaging camera of all bionic quadruped robots;

[0023] determining ambient temperature distribution data of a current scene based on the thermal imaging data, and distinguishing static objects from dynamic objects in the current scene based on the ambient temperature distribution data;

[0024] Performing four-dimensional tensor compression on the visible light video data and the thermal imaging data to obtain a target four-dimensional block integrating space, spectrum, time and channels, associating the target four-dimensional block with the local submap, and performing stitching correction on the local submap according to a preset stitching algorithm to obtain corrected map data;

[0025] The preset GIS elevation data is called to fill the inspection blind spots of each bionic quadruped robot, and a global map that distinguishes the static objects from the dynamic objects is generated according to the preset GIS elevation data and the corrected map data.

[0026] Optionally, predicting the future motion path of the dynamic obstacle through the LSTM model according to the global map includes:

[0027] Obtaining a historical motion trajectory of a dynamic obstacle within a preset historical timing period according to the global map;

[0028] The historical motion trajectory is input into the LSTM model to obtain the future motion prediction trajectory of the preset prediction timing period, and the future motion prediction trajectory is used as the future motion trajectory.

[0029] Optionally, determining a planned inspection path according to the future motion path, performing real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data, and obtaining a real-scene mapping optimization result after virtual-real synchronization includes:

[0030] Obtaining current inspection requirements, and determining a planned inspection path for each bionic quadruped robot based on the future motion path and the current inspection requirements;

[0031] Acquire real-time video data, and use OpenGL or WebGL to overlay the real-time video data with the planned inspection path to achieve real-time dynamic 3D mapping and obtain a 3D mapping result;

[0032] Real-time ray tracing and edge processing are performed on the three-dimensional mapping result, and accelerated rendering is performed on the edge processing result to obtain a real-scene mapping optimization result after virtual and real synchronization.

[0033] In a second aspect, to achieve the above-mentioned purpose, the present invention further proposes a three-dimensional dynamic real scene mapping optimization device, the three-dimensional dynamic real scene mapping optimization device comprising:

[0034] A map generation module is used to start all bionic quadruped robots, obtain inspection data of all bionic quadruped robots, and generate a global map of the current scene based on the inspection data;

[0035] A path prediction module, configured to predict the future movement path of a dynamic obstacle using an LSTM model based on the global map;

[0036] The real-scene mapping optimization module is used to determine the planned inspection path according to the future motion path, perform real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data, and obtain the real-scene mapping optimization result after virtual and real synchronization.

[0037] In a third aspect, to achieve the above-mentioned objectives, the present invention further proposes a three-dimensional dynamic real scene mapping optimization device, the three-dimensional dynamic real scene mapping optimization device comprising: a memory, a processor, and a three-dimensional dynamic real scene mapping optimization program stored in the memory and executable on the processor, the three-dimensional dynamic real scene mapping optimization program being configured to implement the steps of the three-dimensional dynamic real scene mapping optimization method described above.

[0038] In a fourth aspect, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a three-dimensional dynamic real scene mapping optimization program is stored. When the three-dimensional dynamic real scene mapping optimization program is executed by a processor, the steps of the three-dimensional dynamic real scene mapping optimization method described above are implemented.

[0039] The three-dimensional dynamic real-scene mapping optimization method proposed in the present invention starts all bionic quadruped robots, obtains inspection data of all bionic quadruped robots, and generates a global map of the current scene based on the inspection data; predicts the future movement path of dynamic obstacles through an LSTM model based on the global map; determines a planned inspection path based on the future movement path, and performs real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data to obtain a real-scene mapping optimization result after virtual and real synchronization; the method can timely reflect equipment status and environmental changes, improve the timeliness and safety of inspection decisions, reduce rendering delays, improve positioning accuracy in complex environments, accurately map dynamic changes on site, and improve the speed and efficiency of three-dimensional dynamic real-scene mapping optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0041] Figure 2 This is a flow chart of a first embodiment of a three-dimensional dynamic real scene mapping optimization method according to the present invention;

[0042] Figure 3 This is a flow chart of a second embodiment of the three-dimensional dynamic real scene mapping optimization method of the present invention;

[0043] Figure 4 This is a flow chart of a third embodiment of the three-dimensional dynamic real scene mapping optimization method of the present invention;

[0044] Figure 5 This is a flow chart of a fourth embodiment of a three-dimensional dynamic real scene mapping optimization method according to the present invention;

[0045] Figure 6 This is a functional module diagram of the first embodiment of the 3D dynamic real scene mapping optimization device of the present invention.

[0046] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] The solution of the embodiment of the present invention is mainly: by starting all bionic quadruped robots, obtaining the inspection data of all bionic quadruped robots, and generating a global map of the current scene according to the inspection data; predicting the future movement path of dynamic obstacles through the LSTM model according to the global map; determining the planned inspection path according to the future movement path, and performing real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data to obtain the real-scene mapping optimization result after virtual and real synchronization; it can timely reflect the equipment status and environmental changes, improve the timeliness and safety of inspection decisions, reduce rendering delays, improve positioning accuracy in complex environments, accurately map dynamic changes on site, and improve the speed and efficiency of three-dimensional dynamic real-scene mapping optimization, and solve the technical problems in the existing technology that the digital twin platform cannot timely reflect the equipment status and environmental changes, the timeliness and safety of inspection decisions are poor, and there are rendering delays and insufficient accuracy.

[0049] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0050] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a stable memory (Non-Volatile Memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0051] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0052] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating device, a network communication module, a user interface module, and a three-dimensional dynamic real scene mapping optimization program.

[0053] The device of the present invention calls the three-dimensional dynamic real scene mapping optimization program stored in the memory 1005 through the processor 1001 and performs the following operations:

[0054] Starting all bionic quadruped robots, obtaining inspection data of all bionic quadruped robots, and generating a global map of the current scene based on the inspection data;

[0055] Predicting the future movement path of the dynamic obstacle through the LSTM model based on the global map;

[0056] A planned inspection path is determined according to the future motion path, and real-time dynamic three-dimensional mapping is performed based on the planned inspection path in combination with real-time video data to obtain a real-scene mapping optimization result after virtual-real synchronization.

[0057] The device of the present invention calls the three-dimensional dynamic real scene mapping optimization program stored in the memory 1005 through the processor 1001, and further performs the following operations:

[0058] Starting all bionic quadruped robots, and aligning clocks of all sensors in all bionic quadruped robots using the Precision Time Protocol (PTP);

[0059] Acquire the laser radar point cloud data and inertial measurement unit (IMU) pose data of each bionic quadruped robot after clock alignment, combine the laser radar point cloud data and the IMU pose data to construct a local submap;

[0060] Obtain visible light video data and thermal imaging data of each bionic quadruped robot after clock alignment, perform splicing correction on the local submaps according to the visible light video data and thermal imaging data, and generate a global map of the current scene.

[0061] The device of the present invention calls the three-dimensional dynamic real scene mapping optimization program stored in the memory 1005 through the processor 1001, and further performs the following operations:

[0062] Starting all bionic quadruped robots, and aligning the data timestamps of the lidar, inertial measurement unit (IMU), and dual-spectral camera in all bionic quadruped robots through the Precision Time Protocol (PTP) for synchronization;

[0063] The spatial coordinates of each sensor are calibrated offline using a checkerboard calibration method to align the 3D point cloud of the lidar with the image captured by the dual-spectrum camera in space.

[0064] The device of the present invention calls the three-dimensional dynamic real scene mapping optimization program stored in the memory 1005 through the processor 1001, and further performs the following operations:

[0065] Scanning the ground features of the current scene using the laser radars of all the bionic quadruped robots to obtain laser radar point cloud data;

[0066] The inertial measurement unit (IMU) of each of the bionic quadruped robots measures the robot's motion speed and angle in real time, and generates IMU pose data based on the motion speed and angle.

[0067] The laser radar point cloud data and the IMU pose data of each bionic quadruped robot are combined to construct a local submap corresponding to each bionic quadruped robot.

[0068] The device of the present invention calls the three-dimensional dynamic real scene mapping optimization program stored in the memory 1005 through the processor 1001, and further performs the following operations:

[0069] Obtain visible light video data and thermal imaging data through the dual-spectral imaging camera of all bionic quadruped robots;

[0070] determining ambient temperature distribution data of a current scene based on the thermal imaging data, and distinguishing static objects from dynamic objects in the current scene based on the ambient temperature distribution data;

[0071] Performing four-dimensional tensor compression on the visible light video data and the thermal imaging data to obtain a target four-dimensional block integrating space, spectrum, time and channels, associating the target four-dimensional block with the local submap, and performing stitching correction on the local submap according to a preset stitching algorithm to obtain corrected map data;

[0072] The preset GIS elevation data is called to fill the inspection blind spots of each bionic quadruped robot, and a global map that distinguishes the static objects from the dynamic objects is generated according to the preset GIS elevation data and the corrected map data.

[0073] The device of the present invention calls the three-dimensional dynamic real scene mapping optimization program stored in the memory 1005 through the processor 1001, and further performs the following operations:

[0074] Obtaining a historical motion trajectory of a dynamic obstacle within a preset historical timing period according to the global map;

[0075] The historical motion trajectory is input into the LSTM model to obtain the future motion prediction trajectory of the preset prediction timing period, and the future motion prediction trajectory is used as the future motion trajectory.

[0076] The device of the present invention calls the three-dimensional dynamic real scene mapping optimization program stored in the memory 1005 through the processor 1001, and further performs the following operations:

[0077] Obtaining current inspection requirements, and determining a planned inspection path for each bionic quadruped robot based on the future motion path and the current inspection requirements;

[0078] Acquire real-time video data, and use OpenGL or WebGL to overlay the real-time video data with the planned inspection path to achieve real-time dynamic 3D mapping and obtain a 3D mapping result;

[0079] Real-time ray tracing and edge processing are performed on the three-dimensional mapping result, and accelerated rendering is performed on the edge processing result to obtain a real-scene mapping optimization result after virtual and real synchronization.

[0080] Through the above scheme, this embodiment starts all bionic quadruped robots, obtains the inspection data of all bionic quadruped robots, and generates a global map of the current scene based on the inspection data; predicts the future movement path of dynamic obstacles through the LSTM model based on the global map; determines the planned inspection path based on the future movement path, and performs real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data to obtain a real-scene mapping optimization result after virtual and real synchronization; it can timely reflect equipment status and environmental changes, improve the timeliness and safety of inspection decisions, reduce rendering delays, improve positioning accuracy in complex environments, can accurately map dynamic changes on site, and improve the speed and efficiency of three-dimensional dynamic real-scene mapping optimization.

[0081] Based on the above hardware structure, an embodiment of the three-dimensional dynamic real scene mapping optimization method of the present invention is proposed.

[0082] Reference Figure 2 , Figure 2 2 is a flow chart of the first embodiment of the 3D dynamic real scene mapping optimization method of the present invention.

[0083] In a first embodiment, the three-dimensional dynamic real scene mapping optimization method includes the following steps:

[0084] Step S10: start all bionic quadruped robots, obtain inspection data of all bionic quadruped robots, and generate a global map of the current scene based on the inspection data.

[0085] It should be noted that after all the bionic quadruped robots in the current scene are started, the bionic quadruped robots can be used to inspect the current scene to obtain inspection data, and a global map of the current scene can be generated based on the inspection data.

[0086] Step S20: Predict the future movement path of the dynamic obstacle through the LSTM model according to the global map.

[0087] It should be understood that the global map can be used to predict the future movement path of the dynamic obstacle in the current scene within a period of time in the future through a Long Short Term Memory (LSTM) model.

[0088] Step S30: determining a planned inspection path according to the future motion path, performing real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data, and obtaining a real-scene mapping optimization result after virtual-real synchronization.

[0089] It can be understood that the inspection paths of all the bionic quadruped robots can be planned based on the future motion path to obtain the planned inspection path. Real-time dynamic three-dimensional mapping can be performed based on the planned inspection path combined with the real-time video data. After virtual-real synchronization, the real-scene mapping optimization result can be obtained.

[0090] Through the above scheme, this embodiment starts all bionic quadruped robots, obtains the inspection data of all bionic quadruped robots, and generates a global map of the current scene based on the inspection data; predicts the future movement path of dynamic obstacles through the LSTM model based on the global map; determines the planned inspection path based on the future movement path, and performs real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data to obtain a real-scene mapping optimization result after virtual and real synchronization; it can timely reflect equipment status and environmental changes, improve the timeliness and safety of inspection decisions, reduce rendering delays, improve positioning accuracy in complex environments, can accurately map dynamic changes on site, and improve the speed and efficiency of three-dimensional dynamic real-scene mapping optimization.

[0091] Further, Figure 3 This is a flow chart of the second embodiment of the three-dimensional dynamic real scene mapping optimization method of the present invention. Figure 3 As shown, a second embodiment of the three-dimensional dynamic real scene mapping optimization method of the present invention is proposed based on the first embodiment. In this embodiment, step S10 specifically includes the following steps:

[0092] Step S11: start all bionic quadruped robots and align the clocks of all sensors in all bionic quadruped robots using the Precision Time Protocol (PTP).

[0093] It should be noted that before using a bionic quadruped robot to conduct environmental perception inspections, the robot needs to be started. After multiple bionic quadruped robots are started, the clocks of all sensors in all bionic quadruped robots can be aligned through the Precision Time Protocol (PTP) protocol (similar to "network time synchronization").

[0094] Furthermore, the step S11 specifically includes the following steps:

[0095] Starting all bionic quadruped robots, and aligning the data timestamps of the lidar, inertial measurement unit (IMU), and dual-spectral camera in all bionic quadruped robots through the Precision Time Protocol (PTP) for synchronization;

[0096] The spatial coordinates of each sensor are calibrated offline using a checkerboard calibration method to align the 3D point cloud of the lidar with the image captured by the dual-spectrum camera in space.

[0097] It can be understood that the clocks of all sensors are aligned through PTP, which ensures that the data timestamps of the lidar, IMU (inertial measurement unit), and dual-spectral camera (visible light + thermal imaging) are completely synchronized (error < 20 microseconds); then, the spatial coordinates of the sensors are calibrated offline through the checkerboard calibration method (error < 5 cm) to ensure that the 3D point cloud of the lidar is spatially aligned with the image captured by the dual-spectral camera.

[0098] Step S12: Obtain the laser radar point cloud data and inertial measurement unit (IMU) pose data of each bionic quadruped robot after clock alignment, combine the laser radar point cloud data and the IMU pose data, and construct a local submap.

[0099] It can be understood that after obtaining the lidar point cloud data and inertial measurement unit (IMU) posture data of each bionic quadruped robot after the clock is aligned, the lidar point cloud data and the IMU posture data can be combined to construct a local sub-map corresponding to each bionic quadruped robot.

[0100] Furthermore, the step S12 specifically includes the following steps:

[0101] Scanning the ground features of the current scene using the laser radars of all the bionic quadruped robots to obtain laser radar point cloud data;

[0102] The inertial measurement unit (IMU) of each of the bionic quadruped robots measures the robot's motion speed and angle in real time, and generates IMU pose data based on the motion speed and angle.

[0103] The laser radar point cloud data and the IMU pose data of each bionic quadruped robot are combined to construct a local submap corresponding to each bionic quadruped robot.

[0104] It should be noted that terrain recognition can be performed through the bionic quadruped robot, and the bionic quadruped robot can choose different gaits according to different terrains; the robot uses lidar to scan ground features (such as the inclination angle of stairs and the height of pipes), and combines thermal imaging cameras to detect the ambient temperature distribution (for example, identifying high-temperature leak points); according to the complexity of the terrain, the robot automatically switches gaits: flat ground: adopts a low-power diagonal gait to save energy; stairs or pipes: switch to a high-leg gait to increase joint torque, and adjust the foothold in real time through the foot-end force sensor to prevent rollover.

[0105] In the specific implementation, when the robot moves, the lidar generates hundreds of thousands of point cloud data per second; the IMU measures the robot's movement speed and angle in real time, and eliminates its own motion errors through IMU pre-integration (calculating posture changes every 0.1 second); then, the speed threshold method (dynamic point cloud speed ≥ 0.2m / s) is used to distinguish between static objects (such as pipes, walls) and dynamic objects (such as moving inspection personnel); dynamic point clouds are marked as low weight to avoid interfering with map construction.

[0106] Step S13: Obtain visible light video data and thermal imaging data of each bionic quadruped robot after clock alignment, perform stitching and correction on the local submaps according to the visible light video data and thermal imaging data, and generate a global map of the current scene.

[0107] It should be understood that after obtaining the visible light video data and thermal imaging data of each bionic quadruped robot after aligning the clock, the local sub-map can be spliced and corrected according to the visible light video data and thermal imaging data to generate a global map of the current scene.

[0108] Furthermore, the step S13 specifically includes the following steps:

[0109] Obtain visible light video data and thermal imaging data through the dual-spectral imaging camera of all bionic quadruped robots;

[0110] determining ambient temperature distribution data of a current scene based on the thermal imaging data, and distinguishing static objects from dynamic objects in the current scene based on the ambient temperature distribution data;

[0111] Performing four-dimensional tensor compression on the visible light video data and the thermal imaging data to obtain a target four-dimensional block integrating space, spectrum, time and channels, associating the target four-dimensional block with the local submap, and performing stitching correction on the local submap according to a preset stitching algorithm to obtain corrected map data;

[0112] The preset GIS elevation data is called to fill the inspection blind spots of each bionic quadruped robot, and a global map that distinguishes the static objects from the dynamic objects is generated according to the preset GIS elevation data and the corrected map data.

[0113] It is understandable that a single robot combines the processed lidar point cloud with the IMU pose data to build a local sub-map; the dynamic point cloud is sparse (only the outline is retained), and the static point cloud is used for accurate modeling; for example, when a dynamic object moves, its trajectory will be marked as a temporary obstacle and will not participate in the global map update.

[0114] In the specific implementation, all robots upload local sub-maps to the central system via wireless networks; the iSAM2 algorithm (an intelligent puzzle tool) can be used to integrate data from multiple machines and correct map stitching errors in real time (for example, when the position deviation of two robots in the same area is greater than 15 cm, they are automatically aligned and corrected); finally, a global map is generated with an error of less than 10 cm (about the width of a palm).

[0115] It should be understood that Geographic Information System (GIS) data can be used to fill in the bionic quadruped robot's inspection blind spots. LiDAR may miss areas on roofs or under pipes. The system calls pre-stored GIS elevation data (similar to a 3D terrain database) to fill in blind spots and ensure map integrity. For example, the high-temperature area of a pipeline leak will be captured by a thermal imaging camera and the location will be accurately marked using GIS data.

[0116] In the specific implementation, the dual-spectral data needs to be aligned and compressed. The specific process is as follows: the dual-spectral camera simultaneously shoots 4K visible light video and thermal imaging images; through four-dimensional tensor compression technology, the two types of data are integrated into a four-dimensional block of "space × spectrum × time × channel", and an asymmetric quantization strategy is adopted to retain the temperature gradient information of the thermal imaging, and each 0.5°C is quantized into a step size (8-bit precision).

[0117] For visible light, adaptive DCT compression (similar to JPEG but more efficient) is used to enhance high-frequency texture details. Specifically, each frame of the image is divided into 8×8 pixel blocks, and the pixel values are converted into frequency coefficients through discrete cosine transform. For low frequencies, their coefficients are retained, and for high frequencies, the high-frequency coefficients are weakened through a quantization table to reduce the amount of data. The data volume is reduced and processed in real time through a Field Programmable Gate Array (FPGA) chip (dedicated circuit), compressing the delay to seconds.

[0118] Traditional Discrete Cosine Transform (DCT) generally uses a fixed 8x8 for image compression. Therefore, for regions with sharp edges, the 8x8 block will lose high-frequency information, and for large smooth regions, the 8x8 block will cause data redundancy. So in parts with more details, small blocks are needed for processing, and large blocks can be used in smooth parts to improve the compression ratio.

[0119] The solution of this scheme is: variable-block DCT

[0120] As the name implies, according to the complexity of the image region, the block size of the DCT transform is adaptively adjusted. In high-detail regions, small blocks (4x4, 8x8) are used to enhance information retention; in low-complexity regions, large blocks (16x16, 32x32) are used to improve the compression ratio.

[0121] The specific implementation process is as follows

[0122] (1) Edge detection + high-frequency calculation

[0123] Use the Sobel operator to detect edges and calculate the local high-frequency energy:

[0124]

[0125] Among them, E(x, y) is the total energy of the pixel at coordinates (x, y), which is accumulated by the energy in the horizontal direction and the vertical direction (as well as the energy contributions in other directions within the neighborhood), reflecting the overall intensity change degree within the neighborhood of this pixel. I(x, y) is the pixel intensity value (such as the grayscale value) at coordinates (x, y), serving as the benchmark for calculating the energy in the horizontal and vertical directions; I(x + i, y + j) is the pixel intensity value at coordinates (x + i, y + j), used to compare with the central pixel I(x, y) to calculate the intensity difference in the horizontal or vertical direction.

[0126] The high-frequency energy determines the DCT block size

[0127] When the energy E < T1, that is, in the smooth region, a larger (16x16) block size is used

[0128] When T1 < E < T2, a medium-sized block (8x8) is used

[0129] When the energy E > T2, that is, in the region with rich details, a smaller block (4x4) is used (T1 and T2 are user-defined values).

[0130] (2) Adaptive block division

[0131] Calculate the high-frequency energy histogram of the image and adjust the DCT block size according to the energy gradient

[0132] (3) Block DCT transform

[0133] Perform DCT transform according to the size of the selected block (orthogonal normalization is performed as follows)

[0134]

[0135] Where F(u, v) is the coefficient at the frequency domain coordinate (u, v) after discrete cosine transform (DCT), which is used to describe the energy distribution of the image at that frequency component; α(u) and α(v) are normalization factors that ensure the orthogonality and energy conservation of the transform; all pixels in the N×N image block are traversed and summed, (x, y) is the spatial domain pixel coordinate (0≤x, y≤N-1); I(x, y) is the pixel intensity value (such as grayscale value) at the coordinate (x, y), which is the original input of the transform; and It is the basis function of DCT, which is used to convert the spatial domain pixel value I(x,y) to the frequency domain; where u and v determine the frequency components, x and y are the current pixel positions, and the frequency characteristics of the cosine function are used to separate the information of different frequencies.

[0136] (4) Entropy coding - using improved adaptive Huffman coding

[0137] In the low-frequency area, standard Huffman coding is used, and in the medium-frequency area, adaptive coding is used, which is optimized based on historical data. In the high-frequency area, the code table is updated using block statistics. After counting the DCT distribution coefficients and grading the energy, the DCT distribution coefficients of the most recent 100 blocks are calculated, and the Huffman tree is dynamically updated. The code length in the low-frequency area is fixed, and the code table in the high-frequency area is retrained.

[0138] CNN is then used for entropy modeling, with the DCT coefficient matrix as input, a 3x3 convolutional layer, ReLU+pooling, and PixelCNN used to model the autoregressive distribution. The probability distribution of each DCT coefficient is calculated to obtain the optimal quantization step size. The optimal coding strategy is selected based on the predicted probability distribution, that is, Huffman coding is used for high confidence intervals, and arithmetic coding is used for low confidence intervals.

[0139] The compressed DCT data stream needs to be inversely transformed to restore the visual effect.

[0140] 2.2 Intelligent quantification of thermal imaging data.

[0141] 2.2.1 Traditional Issues

[0142] Fixed step size quantization (such as 0.5 degrees Celsius / step) has insufficient quantization accuracy when the temperature difference is large, and redundant quantization when the temperature difference is small, wasting storage space.

[0143] 2.2.2 Solution: Adaptive Step Size Quantization

[0144] The CNN is used to adaptively adjust the quantization step size. In the low temperature difference region (temperature change < 2 degrees Celsius), the step size is increased to reduce the amount of data. In the high temperature region (temperature change > 5 degrees Celsius), the step size is reduced to improve the accuracy. Finally, the temperature gradient histogram analysis is used to determine the optimal quantization strategy.

[0145] 2.2.3 Specific Implementation:

[0146] Input 16-bit thermal imaging data.

[0147] Calculate the temperature gradient histogram and count the pixel temperature changes:

[0148] G(x, y) = |T(x + 1, y) - T(x, y)| + |T(x, y + 1) - T(x, y)|

[0149] Where G(x, y) is the temperature gradient at the coordinate (x, y), comprehensively measuring the temperature change amplitude in the horizontal and vertical directions at this point. T(x, y) is the temperature value at the coordinate (x, y), T(x + 1, y) is the temperature value at the coordinate (x + 1, y), used to calculate the temperature difference between adjacent pixels in the horizontal direction; |T(x + 1, y) - T(x, y)| is the absolute value of the temperature difference between adjacent pixels in the horizontal direction, reflecting the temperature change degree in the horizontal direction; T(x, y + 1) is the temperature value at the coordinate (x, y + 1), used to calculate the temperature difference between adjacent pixels in the vertical direction.

[0150] Determine the quantization step size according to the gradient magnitude. When the low gradient G < 5, the quantization step size is 1; when 5 < G < 20, the quantization step size is 0.5; when the high gradient G > 20, the quantization step size is 0.1; the above units are all in degrees Celsius.

[0151] Use CNN to predict the optimal quantization step size and use LSTM to predict the future frame temperature trend.

[0152] CNN predicts the optimal quantization step size. Here, the input is the DCT coefficients of a frame of image, and the output is the optimal quantization step size; CNN adopts a lightweight quantization structure, uses 2 3x3 convolutional layers to extract local patterns, and uses a fully connected layer to classify the optimal quantization step size.

[0153] LSTM predicts the future temperature trend. The input is the average temperature of the past N frames (or the temperature distribution histogram), and the output is the trend of the future T frames. The LSTM network architecture contains 2 layers of LSTM units and 1 layer of fully connected layer to predict the change of future frames, combined with DCT transform to optimize storage.

[0154] This solution utilizes an FPGA+GPU heterogeneous computing architecture. The FPGA is responsible for DCT compression and entropy coding to reduce data volume, while the Graphics Processing Unit (GPU) is responsible for Convolutional Neural Network (CNN) calculations and optimizes quantization strategies. These two systems work together to improve computing efficiency and reduce power consumption.

[0155] The specific implementation process is:

[0156] FPGA part: DCT module, based on parallel pipeline acceleration calculation;

[0157] DMA data transmission->FPGA compression and direct transmission to GPU;

[0158] GPU implementation: CNN is used for temperature quantization optimization, LSTM is used to predict temperature trends, and CUDA is used to optimize DCT calculations.

[0159] In the specific implementation, the detailed process of dual-spectrum data alignment and compression is as follows:

[0160] The dual-spectral camera simultaneously captures 4K visible light video and thermal images, merging and compressing the data using frame-by-frame alignment. To improve storage efficiency and preserve key information, this embodiment employs four-dimensional tensor compression technology, organizing the dual-spectral data into a "space × spectrum × time × channel" format. Furthermore, an asymmetric quantization strategy and adaptive DCT compression scheme are designed to address the different characteristics of thermal and visible light data.

[0161] 1. Data organization - the specific process of constructing a four-dimensional tensor:

[0162] The visible light frame Iv(t) and thermal imaging frame It(t) collected by the dual-spectrum camera at time t need to be aligned in space and time, making it difficult to form a unified data structure:

[0163] Spatial dimension (HxW): 4K resolution visible light and thermal imaging, which require interpolation and alignment;

[0164] Spectral dimension: including visible light and thermal imaging, Xt = [Iv, It];

[0165] Time dimension T: stores multiple consecutive frames to facilitate subsequent related time compression;

[0166] Channel dimension C: visible light frames are RGB, and thermal imaging frames are 1-channel temperature data;

[0167] Four-dimensional tensor X = (HxWxTxC).

[0168] 2. Specific process of data alignment:

[0169] Since the resolution, focal length, and optical axis of thermal imaging and visible light may be different, a spatial transformation of the thermal imaging is required to align the visible light:

[0170] Use a checkerboard to calibrate the camera's internal and external parameters, solve the homography transformation moment H between the two, and use bilinear interpolation to convert the thermal imaging frame to 4k size to align it with the visible light pixel level: I' t =H*I t ,Afterwards, edge detection is performed by the Sobel operator, and a ,comparison adjustment is performed at the edge of the thermal ,imaging so that the two match each other.

[0171] 3. Specific process of asymmetric quantification:

[0172] Without affecting accuracy, it can reduce data volume and improve storage and transmission efficiency. Quantification of thermal imaging data.

[0173] Non-uniform quantization is used with a step size of 0.5.

[0174]

[0175] Among them, Q(I't) is the result of quantifying the thermal imaging temperature value (I't), I' t is the thermal imaging temperature value at the current moment, T min is the lowest temperature value in the thermal imaging data.

[0176] The local histogram is equalized to ensure good contrast between dark and bright areas. LSTM is used to predict future frame temperature trends and adaptively quantize the mutation area.

[0177] DCT transform + adaptive quantization, step 1, block processing, divide the image into 8x8 small blocks, perform DCT transform on each block, step 2, retain low frequencies to ensure the integrity of the main image information, and use CNN adaptive quantization step size for high-frequency areas:

[0178]

[0179] Among them, Q(u,v) is the quantized frequency domain coefficient, F(u,v) is the coefficient at the frequency domain coordinate (u,v) after discrete cosine transform (DCT), which reflects the energy distribution of the image at this frequency component, and CNN(F) is the adaptive quantization step size output by the convolutional neural network (CNN) after processing the DCT coefficient F.

[0180] Step three: entropy coding is used to further compress the data.

[0181] 4.FPGA acceleration and real-time processing

[0182] To meet real-time requirements, the entire data compression process is accelerated by FPGA, significantly reducing latency:

[0183] (1) Parallelization of DCT changes: 8x8 DCT parallel operations are achieved through a pipeline architecture to improve computing efficiency.

[0184] (2) CNN quantization step size prediction hardware implementation uses a low-power CNN model and converts it into a format that can be run on FPGA.

[0185] (3) Implement time series analysis in FPGA and adaptively adjust the quantization step size.

[0186] 5. The total data flow is as follows

[0187] Input: 4K visible light + thermal imaging frames

[0188] Data alignment:

[0189] Spatial transformation (homography)

[0190] Interpolation Alignment

[0191] Thermal imaging data processing:

[0192] 0.5℃ quantization (8bit)

[0193] LSTM predicts future frames and adjusts quantization step size

[0194] Visible light data compression:

[0195] 8×8 DCT transform

[0196] CNN quantization step selection

[0197] Entropy coding (Huffman / arithmetic)

[0198] FPGA real-time processing

[0199] Output: Optimized bispectral data stream

[0200] This embodiment uses homography transformation (Homography) + super-resolution interpolation to perform high-precision alignment; homography transformation solves the problems of lens distortion and parallax mismatch, so that the two data are accurately aligned at the pixel level; super-resolution interpolation (based on deep learning) solves the blur problem of low-resolution thermal imaging and improves temperature details.

[0201] This embodiment adopts a non-uniform quantization strategy (0.5°C step size, 8-bit storage): it improves the quantization accuracy of temperature data to ±0.5°C, which is much higher than traditional methods. LSTM predicts future frame temperature trends and adaptively adjusts the quantization step size in sudden change areas to ensure complete temperature difference information.

[0202] This embodiment uses CNN to predict the optimal quantization step size to achieve adaptive DCT compression: deep learning is used to analyze the high-frequency content of the image, and the DCT quantization table is dynamically adjusted to retain key details. For low-contrast areas, the storage accuracy of low-frequency components is improved to ensure visual clarity. Entropy coding (Huffman / arithmetic coding) optimizes storage, reducing the amount of data while retaining key information.

[0203] This embodiment uses FPGA hardware acceleration: DCT transform is performed through 8×8 parallel computing, which greatly improves processing speed; CNN quantization step prediction is deployed in the FPGA-DPU (Deep Learning Processing Unit), achieving millisecond-level inference; LSTM temperature trend prediction is accelerated by the FPGA RNN unit, achieving real-time dynamic adjustment.

[0204] This embodiment uses the above scheme to start all bionic quadruped robots and align the clocks of all sensors in all bionic quadruped robots through the precise time protocol PTP; obtain the lidar point cloud data and inertial measurement unit IMU posture data of each bionic quadruped robot after the clock is aligned, combine the lidar point cloud data and the IMU posture data to construct a local submap; obtain the visible light video data and thermal imaging data of each bionic quadruped robot after the clock is aligned, splice and correct the local submap according to the visible light video data and thermal imaging data, and generate a global map of the current scene, which can improve the real-time mapping accuracy and enhance the robustness and real-time performance of three-dimensional modeling.

[0205] Further, Figure 4 This is a flow chart of the third embodiment of the three-dimensional dynamic real scene mapping optimization method of the present invention. Figure 4 As shown, a third embodiment of the three-dimensional dynamic real scene mapping optimization method of the present invention is proposed based on the first embodiment. In this embodiment, step S20 specifically includes the following steps:

[0206] Step S21: Obtain the historical motion trajectory of the dynamic obstacle within a preset historical timing period according to the global map.

[0207] It should be noted that the historical movement trajectory of the dynamic obstacle within a preset historical timing period can be obtained according to the global map.

[0208] Step S22: Input the historical motion trajectory into the LSTM model to obtain the future motion prediction trajectory of the preset prediction timing period, and use the future motion prediction trajectory as the future motion trajectory.

[0209] It can be understood that by inputting the historical motion trajectory into the LSTM model, the future motion prediction trajectory of the preset prediction timing period is obtained, and the future motion prediction trajectory is used as the future motion trajectory.

[0210] In its specific implementation, the system uses the LSTM model (a neural network that can learn temporal patterns) to predict the future path of a dynamic obstacle; it inputs the obstacle's motion trajectory over the past 5 seconds and outputs the predicted path for the next 5 seconds (with an error of <0.5 meters).

[0211] Through the above scheme, this embodiment obtains the historical motion trajectory of the dynamic obstacle within a preset historical timing period based on the global map; inputs the historical motion trajectory into the LSTM model to obtain the future motion prediction trajectory of the preset prediction timing period, and uses the future motion prediction trajectory as the future motion trajectory; it can timely reflect equipment status and environmental changes, thereby improving the timeliness and safety of inspection decisions.

[0212] Further, Figure 5 FIG. 4 is a flow chart of a fourth embodiment of a three-dimensional dynamic real scene mapping optimization method according to the present invention. Figure 5 As shown, a fourth embodiment of the three-dimensional dynamic real scene mapping optimization method of the present invention is proposed based on the first embodiment. In this embodiment, step S30 specifically includes the following steps:

[0213] Step S31 : obtaining the current inspection demand, and determining the planned inspection path of each bionic quadruped robot according to the future motion path and the current inspection demand.

[0214] It should be noted that after obtaining the current inspection demand, the planned inspection path of each bionic quadruped robot for the current scene can be determined according to the future motion path and the current inspection demand.

[0215] Step S32: Acquire real-time video data, and use OpenGL or WebGL to superimpose the real-time video data with the planned inspection path to achieve real-time dynamic three-dimensional mapping and obtain a three-dimensional mapping result.

[0216] It is understandable that after acquiring the real-time video data, OpenGL or WebGL can be used to superimpose the real-time video data with the planned inspection path to achieve real-time dynamic three-dimensional mapping and obtain a three-dimensional mapping result.

[0217] In the specific implementation, the use of OpenGL / WebGL can achieve the superposition of real-time video and 3D models, and combined with the NVIDIA Omniverse platform to introduce real-time ray tracing technology and edge AI processing, using RTX GPU accelerated rendering to achieve virtual and real synchronization (robotic arm movement accuracy can reach ≤0.2mm), providing a high-precision perspective for the "virtual inspector"; by introducing real-time dynamic SLAM (combined with the improved Cartographer framework, IMU pre-integration, factor graph optimization and LSTM time series optimization), real-time and dynamic 3D modeling of the on-site environment is achieved, ensuring that the digital twin maintains 1:1 synchronization with the actual situation; combined with the NVIDIA Omniverse platform, using RTX GPU acceleration and real-time ray tracing technology, and introducing edge AI processing, high-fidelity, real-time virtual and real fusion is achieved, ensuring that the virtual environment can accurately map the dynamic changes on site.

[0218] Step S33: perform real-time ray tracing and edge processing on the three-dimensional mapping result, and perform accelerated rendering on the edge processing result to obtain a real-scene mapping optimization result after virtual-real synchronization.

[0219] It should be understood that after real-time ray tracing and edge processing of the three-dimensional mapping results, the edge processing results can be accelerated to render the real-scene mapping optimization results after virtual and real synchronization. After detecting the global map update, the updated global map can be sent to all robots via the wireless network to guide their next actions.

[0220] In practice, the cloud can be connected to the Industrial Metaverse platform, where compressed bispectral data, global maps, and equipment status information can be uploaded to the Energy Industry Metaverse platform, achieving virtual-real integration. For example, managers can use AR glasses to view real-time 3D maps to quickly locate leaks or plan inspection routes.

[0221] Through the above-mentioned scheme, this embodiment obtains the current inspection demand and determines the planned inspection path of each bionic quadruped robot according to the future motion path and the current inspection demand; obtains real-time video data, and uses OpenGL or WebGL to superimpose the real-time video data with the planned inspection path to achieve real-time dynamic three-dimensional mapping and obtain a three-dimensional mapping result; performs real-time ray tracing and edge processing on the three-dimensional mapping result, and accelerates the rendering of the edge processing result to obtain a real-scene mapping optimization result after virtual-real synchronization, which can timely reflect the equipment status and environmental changes, improve the timeliness and safety of inspection decisions, reduce rendering delays, improve positioning accuracy in complex environments, can accurately map dynamic changes on site, and improve the speed and efficiency of three-dimensional dynamic real-scene mapping optimization.

[0222] Accordingly, the present invention further provides a three-dimensional dynamic real scene mapping optimization device.

[0223] Reference Figure 6 , Figure 6 This is a functional module diagram of the first embodiment of the 3D dynamic real scene mapping optimization device of the present invention.

[0224] In a first embodiment of the three-dimensional dynamic real scene mapping optimization device of the present invention, the three-dimensional dynamic real scene mapping optimization device includes:

[0225] The map generation module 10 is used to start all bionic quadruped robots, obtain inspection data of all bionic quadruped robots, and generate a global map of the current scene according to the inspection data.

[0226] The path prediction module 20 is used to predict the future movement path of the dynamic obstacle through the LSTM model according to the global map.

[0227] The real scene mapping optimization module 30 is used to determine a planned inspection path according to the future motion path, perform real-time dynamic 3D mapping based on the planned inspection path in combination with real-time video data, and obtain a real scene mapping optimization result after virtual and real synchronization.

[0228] The map generation module 10 is also used to start all bionic quadruped robots and align the clocks of all sensors in all bionic quadruped robots through the precise time protocol PTP; obtain the lidar point cloud data and inertial measurement unit IMU posture data of each bionic quadruped robot after the clocks are aligned, combine the lidar point cloud data and the IMU posture data to construct a local sub-map; obtain the visible light video data and thermal imaging data of each bionic quadruped robot after the clocks are aligned, splice and correct the local sub-map according to the visible light video data and thermal imaging data, and generate a global map of the current scene.

[0229] The map generation module 10 is also used to start all bionic quadruped robots, align the data timestamps of the lidar, inertial measurement unit IMU and dual-spectral camera in all bionic quadruped robots through the precise time protocol PTP for synchronization; and calibrate the spatial coordinates of each sensor offline through the checkerboard calibration method to spatially align the 3D point cloud of the lidar with the image captured by the dual-spectral camera.

[0230] The map generation module 10 is also used to scan the ground features of the current scene through the laser radars of all the bionic quadruped robots to obtain laser radar point cloud data; measure the robot movement speed and angle in real time through the inertial measurement units (IMUs) of all the bionic quadruped robots, and generate IMU posture data according to the movement speed and angle; combine the laser radar point cloud data and the IMU posture data of each bionic quadruped robot to construct a local sub-map corresponding to each bionic quadruped robot.

[0231] The map generation module 10 is further used to obtain visible light video data and thermal imaging data through the dual-spectral imaging cameras of all the bionic quadruped robots; determine the ambient temperature distribution data of the current scene based on the thermal imaging data, and distinguish static objects and dynamic objects in the current scene based on the ambient temperature distribution data; perform four-dimensional tensor compression on the visible light video data and the thermal imaging data to obtain a target four-dimensional block that integrates space, spectrum, time and channels, associate the target four-dimensional block with the local sub-map, and perform stitching correction on the local sub-map according to a preset stitching algorithm to obtain corrected map data; call preset GIS elevation data to fill the inspection blind spots of each bionic quadruped robot, and generate a global map that distinguishes between the static objects and the dynamic objects based on the preset GIS elevation data and the corrected map data.

[0232] The path prediction module 20 is further used to obtain the historical motion trajectory of the dynamic obstacle within a preset historical timing period based on the global map; input the historical motion trajectory into the LSTM model to obtain the future motion prediction trajectory of the preset prediction timing period, and use the future motion prediction trajectory as the future motion trajectory.

[0233] The real-scene mapping optimization module 30 is further used to obtain current inspection requirements, determine the planned inspection path of each bionic quadruped robot based on the future motion path and the current inspection requirements; obtain real-time video data, and use OpenGL or WebGL to superimpose the real-time video data with the planned inspection path to achieve real-time dynamic three-dimensional mapping and obtain a three-dimensional mapping result; perform real-time ray tracing and edge processing on the three-dimensional mapping result, and accelerate rendering of the edge processing result to obtain a real-scene mapping optimization result after virtual and real synchronization.

[0234] The steps implemented by the functional modules of the 3D dynamic real scene mapping optimization device may refer to the various embodiments of the 3D dynamic real scene mapping optimization method of the present invention, and will not be repeated here.

[0235] In addition, an embodiment of the present invention further provides a storage medium, wherein a three-dimensional dynamic real scene mapping optimization program is stored on the storage medium. When the three-dimensional dynamic real scene mapping optimization program is executed by a processor, the following operations are performed:

[0236] Starting all bionic quadruped robots, obtaining inspection data of all bionic quadruped robots, and generating a global map of the current scene based on the inspection data;

[0237] Predicting the future movement path of the dynamic obstacle through the LSTM model based on the global map;

[0238] A planned inspection path is determined according to the future motion path, and real-time dynamic three-dimensional mapping is performed based on the planned inspection path in combination with real-time video data to obtain a real-scene mapping optimization result after virtual-real synchronization.

[0239] Furthermore, when the three-dimensional dynamic real scene mapping optimization program is executed by the processor, the following operations are also implemented:

[0240] Starting all bionic quadruped robots, and aligning clocks of all sensors in all bionic quadruped robots using the Precision Time Protocol (PTP);

[0241] Acquire the laser radar point cloud data and inertial measurement unit (IMU) pose data of each bionic quadruped robot after clock alignment, combine the laser radar point cloud data and the IMU pose data to construct a local submap;

[0242] Obtain visible light video data and thermal imaging data of each bionic quadruped robot after clock alignment, perform splicing correction on the local submaps according to the visible light video data and thermal imaging data, and generate a global map of the current scene.

[0243] Furthermore, when the three-dimensional dynamic real scene mapping optimization program is executed by the processor, the following operations are also implemented:

[0244] Starting all bionic quadruped robots, and aligning the data timestamps of the lidar, inertial measurement unit (IMU), and dual-spectral camera in all bionic quadruped robots through the Precision Time Protocol (PTP) for synchronization;

[0245] The spatial coordinates of each sensor are calibrated offline using a checkerboard calibration method to align the 3D point cloud of the lidar with the image captured by the dual-spectrum camera in space.

[0246] Furthermore, when the three-dimensional dynamic real scene mapping optimization program is executed by the processor, the following operations are also implemented:

[0247] Scanning the ground features of the current scene using the laser radars of all the bionic quadruped robots to obtain laser radar point cloud data;

[0248] The inertial measurement unit (IMU) of each of the bionic quadruped robots measures the robot's motion speed and angle in real time, and generates IMU pose data based on the motion speed and angle.

[0249] The laser radar point cloud data and the IMU pose data of each bionic quadruped robot are combined to construct a local submap corresponding to each bionic quadruped robot.

[0250] Furthermore, when the three-dimensional dynamic real scene mapping optimization program is executed by the processor, the following operations are also implemented:

[0251] Obtain visible light video data and thermal imaging data through the dual-spectral imaging camera of all bionic quadruped robots;

[0252] determining ambient temperature distribution data of a current scene based on the thermal imaging data, and distinguishing static objects from dynamic objects in the current scene based on the ambient temperature distribution data;

[0253] Performing four-dimensional tensor compression on the visible light video data and the thermal imaging data to obtain a target four-dimensional block integrating space, spectrum, time and channels, associating the target four-dimensional block with the local submap, and performing stitching correction on the local submap according to a preset stitching algorithm to obtain corrected map data;

[0254] The preset GIS elevation data is called to fill the inspection blind spots of each bionic quadruped robot, and a global map that distinguishes the static objects from the dynamic objects is generated according to the preset GIS elevation data and the corrected map data.

[0255] Furthermore, when the three-dimensional dynamic real scene mapping optimization program is executed by the processor, the following operations are also implemented:

[0256] Obtaining a historical motion trajectory of a dynamic obstacle within a preset historical timing period according to the global map;

[0257] The historical motion trajectory is input into the LSTM model to obtain the future motion prediction trajectory of the preset prediction timing period, and the future motion prediction trajectory is used as the future motion trajectory.

[0258] Furthermore, when the three-dimensional dynamic real scene mapping optimization program is executed by the processor, the following operations are also implemented:

[0259] Obtaining current inspection requirements, and determining a planned inspection path for each bionic quadruped robot based on the future motion path and the current inspection requirements;

[0260] Acquire real-time video data, and use OpenGL or WebGL to overlay the real-time video data with the planned inspection path to achieve real-time dynamic 3D mapping and obtain a 3D mapping result;

[0261] Real-time ray tracing and edge processing are performed on the three-dimensional mapping result, and accelerated rendering is performed on the edge processing result to obtain a real-scene mapping optimization result after virtual and real synchronization.

[0262] Those skilled in the art will understand that all or part of the steps in the above-mentioned implementation method can be completed by instructing related hardware through a program. The program is stored in a storage medium and includes a number of instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application; and the aforementioned storage medium is a computer-readable storage medium, including: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0263] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0264] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0265] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A three-dimensional dynamic real scene mapping optimization method, characterized in that: The three-dimensional dynamic real scene mapping optimization method comprises: Starting all bionic quadruped robots, obtaining inspection data of all bionic quadruped robots, and generating a global map of the current scene based on the inspection data; Predicting the future movement path of the dynamic obstacle through the LSTM model based on the global map; A planned inspection path is determined according to the future motion path, and real-time dynamic three-dimensional mapping is performed based on the planned inspection path in combination with real-time video data to obtain a real-scene mapping optimization result after virtual-real synchronization.

2. The three-dimensional dynamic real scene mapping optimization method according to claim 1, characterized in that: The step of starting all bionic quadruped robots, obtaining inspection data of all bionic quadruped robots, and generating a global map of the current scene based on the inspection data includes: Starting all bionic quadruped robots, and aligning clocks of all sensors in all bionic quadruped robots using the Precision Time Protocol (PTP); Acquire the laser radar point cloud data and inertial measurement unit (IMU) pose data of each bionic quadruped robot after clock alignment, combine the laser radar point cloud data and the IMU pose data to construct a local submap; Obtain visible light video data and thermal imaging data of each bionic quadruped robot after clock alignment, perform splicing correction on the local submaps according to the visible light video data and thermal imaging data, and generate a global map of the current scene.

3. The three-dimensional dynamic real scene mapping optimization method according to claim 2, characterized in that: The step of starting all bionic quadruped robots and aligning clocks of all sensors in all bionic quadruped robots using the Precision Time Protocol (PTP) includes: Starting all bionic quadruped robots, and aligning the data timestamps of the lidar, inertial measurement unit (IMU), and dual-spectral camera in all bionic quadruped robots through the Precision Time Protocol (PTP) for synchronization; The spatial coordinates of each sensor are calibrated offline using a checkerboard calibration method to align the 3D point cloud of the lidar with the image captured by the dual-spectrum camera in space.

4. The three-dimensional dynamic real scene mapping optimization method according to claim 2, characterized in that: The step of obtaining the laser radar point cloud data and the inertial measurement unit (IMU) pose data of each bionic quadruped robot after clock alignment, combining the laser radar point cloud data and the IMU pose data to construct a local submap includes: Scanning the ground features of the current scene using the laser radars of all the bionic quadruped robots to obtain laser radar point cloud data; The inertial measurement unit (IMU) of each of the bionic quadruped robots measures the robot's motion speed and angle in real time, and generates IMU pose data based on the motion speed and angle. The laser radar point cloud data and the IMU pose data of each bionic quadruped robot are combined to construct a local submap corresponding to each bionic quadruped robot.

5. The three-dimensional dynamic real scene mapping optimization method according to claim 2, characterized in that: The step of acquiring the visible light video data and thermal imaging data of each bionic quadruped robot after clock alignment, and performing splicing correction on the local submaps according to the visible light video data and thermal imaging data to generate a global map of the current scene includes: Obtain visible light video data and thermal imaging data through the dual-spectral imaging camera of all bionic quadruped robots; determining ambient temperature distribution data of a current scene based on the thermal imaging data, and distinguishing static objects from dynamic objects in the current scene based on the ambient temperature distribution data; Performing four-dimensional tensor compression on the visible light video data and the thermal imaging data to obtain a target four-dimensional block integrating space, spectrum, time and channels, associating the target four-dimensional block with the local submap, and performing stitching correction on the local submap according to a preset stitching algorithm to obtain corrected map data; The preset GIS elevation data is called to fill the inspection blind spots of each bionic quadruped robot, and a global map that distinguishes the static objects from the dynamic objects is generated according to the preset GIS elevation data and the corrected map data.

6. The three-dimensional dynamic real scene mapping optimization method according to claim 1, characterized in that: The predicting of the future motion path of the dynamic obstacle by using the LSTM model according to the global map includes: Obtaining a historical motion trajectory of a dynamic obstacle within a preset historical timing period according to the global map; The historical motion trajectory is input into the LSTM model to obtain the future motion prediction trajectory of the preset prediction timing period, and the future motion prediction trajectory is used as the future motion trajectory.

7. The three-dimensional dynamic real scene mapping optimization method according to claim 1, characterized in that: The method of determining a planned inspection path according to the future motion path, performing real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data, and obtaining a real-scene mapping optimization result after virtual-real synchronization includes: Obtaining current inspection requirements, and determining a planned inspection path for each bionic quadruped robot based on the future motion path and the current inspection requirements; Acquire real-time video data, and use OpenGL or WebGL to overlay the real-time video data with the planned inspection path to achieve real-time dynamic 3D mapping and obtain a 3D mapping result; Real-time ray tracing and edge processing are performed on the three-dimensional mapping result, and accelerated rendering is performed on the edge processing result to obtain a real-scene mapping optimization result after virtual and real synchronization.

8. A three-dimensional dynamic real scene mapping optimization device, characterized in that: The three-dimensional dynamic real scene mapping optimization device includes: A map generation module is used to start all bionic quadruped robots, obtain inspection data of all bionic quadruped robots, and generate a global map of the current scene based on the inspection data; A path prediction module, configured to predict the future movement path of a dynamic obstacle using an LSTM model based on the global map; The real-scene mapping optimization module is used to determine the planned inspection path according to the future motion path, perform real-time dynamic three-dimensional mapping based on the planned inspection path in combination with real-time video data, and obtain the real-scene mapping optimization result after virtual and real synchronization.

9. A three-dimensional dynamic real scene mapping optimization device, characterized in that: The three-dimensional dynamic real scene mapping optimization device includes: a memory, a processor, and a three-dimensional dynamic real scene mapping optimization program stored in the memory and executable on the processor, wherein the three-dimensional dynamic real scene mapping optimization program is configured to implement the steps of the three-dimensional dynamic real scene mapping optimization method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a three-dimensional dynamic real scene mapping optimization program, which, when executed by a processor, implements the steps of the three-dimensional dynamic real scene mapping optimization method according to any one of claims 1 to 7.

Images