Work guidance method and device for engineering machinery, electronic equipment and storage medium

Abstract

This invention discloses a method, device, electronic device, and storage medium for guiding the operation of construction machinery. The method includes: after operation commences, collecting multi-source data using an integrated fusion sensing terminal that integrates at least binocular vision acquisition, dToF depth measurement, and IMU inertial measurement; fusing the data to generate a near-field 3D point cloud and optimizing the continuous pose of the equipment; combining prior geometric parameters of the equipment and point cloud / image extracted features to calculate the bucket joint angles, key point coordinates, and envelope model, and then constructing a geometric model of the working surface; uniformly registering the end-effector status, working surface, construction, and safety constraint information to the world coordinate system to build a multi-layer semantic model of the operation scene, quantifying multiple spatial relationship parameters, and generating guidance and warning information based on these parameters to achieve a fully closed-loop AR cab-based operation guidance system. This solution achieves precise quantification of the operating environment and equipment status, improving the level of intelligent guidance for construction machinery operations.

Description

Technical Field

[0001] This invention relates to the field of intelligent construction machinery, and in particular to a method, apparatus, electronic device, and storage medium for guiding the operation of construction machinery. Background Technology

[0002] Construction machinery is widely used in earthwork construction, site leveling, engineering layout and other scenarios. The accuracy and safety of its operation directly affect the quality of the project.

[0003] As construction requirements continue to increase, the traditional manual operation mode relies on the operator's experience and is easily affected by obstructed vision, dust interference, and weak texture environment, making it difficult to ensure the consistency of operation and resulting in low operation efficiency.

[0004] Therefore, there is an urgent need for a method to guide the operation of engineering machinery. Summary of the Invention

[0005] This invention provides a method, device, electronic device, storage medium, and computer program product for guiding the operation of engineering machinery.

[0006] According to one aspect of the present invention, a method for guiding the operation of engineering machinery is provided, comprising: When the construction machinery begins operation, it acquires the equipment's prior geometric parameters, pre-stored construction design information and safety constraint information, a 3D point cloud model of the near-field environment, and the continuous pose of the equipment. The 3D point cloud model of the near-field environment and the continuous pose of the equipment are determined and output by an integrated fusion sensing device installed on the construction machinery based on collected multi-source sensing data. The multi-source sensing data includes at least binocular vision data, dToF depth data, and inertial measurement data. Based on the prior geometric parameters of the equipment and the features of the end-effector extracted from continuous 3D point cloud models or binocular vision data, the state information of the end-effector of the construction machinery is calculated; the state information includes joint angles, key point coordinates and envelope model. Based on the continuous 3D point cloud model, the geometric model of the working surface is constructed. The state information of the end mechanism and the geometric model of the working surface are transformed into the world coordinate system by relying on the continuous pose of the equipment. The construction design information and safety constraint information are uniformly registered to the world coordinate system to construct a multi-layer semantic model of the working scene. Based on a multi-layered operational scenario semantic model, spatial relationship parameters between the end-of-operation mechanism and the working surface and the construction design information and safety constraint information are determined. Based on spatial relationship parameters, operation guidance instructions and safety warning information for construction machinery are generated.

[0007] According to another aspect of the present invention, an engineering machinery operation guidance device is provided, comprising: The data acquisition module is used to acquire the prior geometric parameters of the equipment, pre-stored construction design information and safety constraint information, a 3D point cloud model of the near-field environment, and the continuous pose of the equipment when the construction machinery starts operating. The 3D point cloud model of the near-field environment and the continuous pose of the equipment are determined and output by an integrated fusion sensing device installed on the construction machinery based on collected multi-source sensing data. The multi-source sensing data includes at least binocular vision data, dToF depth data, and inertial measurement data. The end-effector state determination module is used to calculate the state information of the end-effector of the construction machinery based on the prior geometric parameters of the equipment and the features of the end-effector extracted from the continuous 3D point cloud model or binocular vision data. The state information includes joint angles, key point coordinates and envelope model. The model building module is used to build a geometric model of the working surface based on a continuous 3D point cloud model. It transforms the state information of the end mechanism and the geometric model of the working surface to the world coordinate system based on the continuous pose of the equipment, and uniformly registers the construction design information and safety constraint information to the world coordinate system, thereby building a multi-layer semantic model of the working scene. The spatial relationship determination module is used to determine the spatial relationship parameters between the end-of-operation mechanism and the working surface and the construction design information and safety constraint information, respectively, based on the multi-layer operation scenario semantic model. The operation guidance module is used to generate operation guidance instructions and safety warning information for construction machinery based on spatial relationship parameters.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to execute the engineering machinery operation guidance method of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the engineering machinery operation guidance method of the present invention.

[0010] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps in the above-described method.

[0011] The technical solution of this invention integrates multi-source data from binocular vision, dToF, inertial and global positioning to generate high-precision continuous three-dimensional point clouds, accurate pose and end state, construct a unified semantic scene model, quantify spatial relationships and output guidance and warnings, so as to realize precise, safe and efficient automated operation guidance for engineering machinery.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1a This is a flowchart illustrating a method for guiding the operation of engineering machinery according to an embodiment of the present invention; Figure 1b This is a schematic diagram of an integrated fusion sensing device provided in an embodiment of the present invention; Figure 1c This is a schematic diagram of deep fusion perception provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating another method for guiding engineering machinery operations provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an engineering machinery operation guidance device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the engineering machinery operation guidance method of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] Example 1 Figure 1aThis is a flowchart of a method for guiding the operation of construction machinery provided in an embodiment of the present invention. This embodiment is applicable to scenarios of high-precision and safe operation guidance of construction machinery in complex construction site environments. The method can be executed by a construction machinery operation guidance device, which can be implemented in hardware and / or software and can be configured in an electronic device.

[0017] like Figure 1a As shown, the methods for guiding the operation of construction machinery include: S101. When the construction machinery begins operation, acquire the equipment's prior geometric parameters, pre-stored construction design information and safety constraint information, a 3D point cloud model of the near-field environment, and the continuous pose of the equipment; wherein, the 3D point cloud model of the near-field environment and the continuous pose of the equipment are determined and output by an integrated fusion sensing device installed on the construction machinery based on the collected multi-source sensing data; the multi-source sensing data includes at least binocular vision data, dToF depth data, and inertial measurement data; S102. Based on the prior geometric parameters of the equipment and the features of the end-effector extracted from the continuous three-dimensional point cloud model or binocular vision data, calculate the state information of the end-effector of the construction machinery; the state information includes joint angles, key point coordinates and envelope model. S103. Construct a geometric model of the working surface based on a continuous three-dimensional point cloud model. Based on the continuous pose of the equipment, convert the state information of the end mechanism and the geometric model of the working surface to the world coordinate system. And uniformly register the construction design information and safety constraint information to the world coordinate system to construct a multi-layer semantic model of the working scene. S104. Based on the multi-layer operation scenario semantic model, determine the spatial relationship parameters between the end-of-operation mechanism and the operation surface and the construction design information and safety constraint information, respectively. S105. Based on spatial relationship parameters, generate operation guidance instructions and safety warning information for construction machinery.

[0018] In step S101, "construction machinery" refers to engineering equipment used in construction scenarios such as excavation, foundation construction, material loading and unloading, engineering layout, slope trimming, and site clearing. It possesses a movable end effector and is capable of performing construction tasks, encompassing various types of construction machinery such as excavators, loaders, scrapers, and multi-joint robotic arm platforms. The construction machinery in this application is pre-installed with an integrated fusion sensing terminal. An exemplary schematic diagram of the integrated fusion sensing device is attached. Figure 1b The device integrates at least the following sensor modules: a binocular vision acquisition module (i.e., Figure 1b The binocular camera), dToF depth measurement module (i.e. Figure 1b dToF depth window in the image), IMU inertial measurement module (such as...) Figure 1b(IMU module). In some embodiments, the terminal also integrates a global positioning module (e.g., IMU module). Figure 1b (RTK module in the middle).

[0019] The integrated fusion sensing device can be installed near the cab, body, robotic arm support, or end effector of construction machinery. Its installation posture is calibrated using extrinsic parameters and then connected to a unified coordinate transformation chain. Simultaneously, unified time synchronization of all sensors is achieved through hardware triggering or a unified timestamp, enabling all modules to work collaboratively within the same hardware platform. Based on this, the system achieves deep fusion of multi-source data, continuous pose estimation, and error constraints through joint state estimation, avoiding the isolated use of information.

[0020] Multi-source sensing data refers to raw observation data from different sensor modalities that are simultaneously collected at the construction machinery operation site, including at least binocular vision data, dToF depth data, and inertial measurement data. If the integrated fusion sensing device also integrates a global positioning module, the multi-source sensing data can also include global positioning data; global positioning data includes at least one of the following: high-precision GNSS positioning data based on RTK, total station measurement data, UWB positioning data, or positioning data obtained from the coordinates of construction control points; The prior geometric parameters of the equipment refer to the mechanical structural parameters that are calibrated and fixed before the construction machinery leaves the factory. These include the boom length, stick length, bucket size, distance between joint axes, joint coordinate system, sensor installation parameters, and the conversion relationship between the equipment coordinate system and the vehicle coordinate system.

[0021] Construction design information refers to engineering construction benchmark data stored in the system in advance, including the three-dimensional model of the design surface, the target construction outline, the preset work path, the elevation benchmark, the slope requirements, the flatness standard, etc.

[0022] Safety constraint information refers to predefined and stored work restricted area boundary data, including underground pipeline ranges, high-voltage cable protection zones, slope hazard zones, personnel restricted areas, equipment restricted areas, and dynamic safety boundaries.

[0023] In this embodiment, when the construction machinery begins operation, a fusion sensing terminal integrated and installed in the cab, vehicle body, or robotic arm support synchronously collects multiple channels of raw data with a unified timestamp: a binocular camera synchronously outputs left and right images at a set frame rate, which are then preprocessed to generate binocular visual data after distortion correction; a dToF depth sensor outputs pixel-level distance measurements at high frequency, which are then processed to generate dToF depth data after noise reduction, multipath suppression, and temperature compensation; an IMU collects triaxial acceleration and triaxial angular velocity at high frequency, which are then processed to generate inertial measurement data after zero-bias correction, temperature drift compensation, and time synchronization alignment; and an RTK module receives satellite signals and differential data in real time, outputting centimeter-level global engineering coordinates, velocity, and attitude angles as global positioning data. Simultaneously, pre-stored prior geometric parameters, construction design information, and safety constraint information are read from local storage or the cloud. All data undergoes timestamp unification, format standardization, abnormal data removal, and data synchronization alignment to ensure spatiotemporal consistency of multi-source data, forming a temporally synchronized, dimensionally complete, and high-quality input dataset.

[0024] The beneficial effects of this step are as follows: through the synchronous acquisition, standardized processing, and unified integration of multi-source heterogeneous data, comprehensive coverage and deep integration of five types of information—near-field perception, global positioning, equipment structure, construction benchmarks, and safety constraints—are achieved. This effectively solves the problems of insufficient information dimensions from a single data source, asynchronous data timing, and inconsistent data quality. It provides data integrity, temporal consistency, and quality reliability guarantees for subsequent high-precision 3D modeling, continuous pose calculation, end-point state analysis, scene semantic construction, spatial relationship quantification, and guidance command generation. This lays the foundation for high precision, stability, and safety in engineering machinery operation guidance from the source.

[0025] Based on the above, in step S101, the process by which the integrated fusion sensing device determines and outputs the 3D point cloud model of the near-field environment and the continuous pose of the device based on the collected multi-source sensing data includes S1-S2: S1. Based on the fusion processing of the binocular vision data and the dToF depth data, obtain a continuous three-dimensional point cloud model of the near-field environment of the operation and the corresponding point cloud fusion confidence.

[0026] S2. While constructing the continuous three-dimensional point cloud model, the binocular vision data, the dToF depth data, and the inertial measurement data are simultaneously fused and jointly optimized to determine the continuous pose of the device.

[0027] In step S1, the near-field environment of the operation refers to the local working space within a certain range around the end of the construction machinery (such as the bucket of an excavator), which directly participates in the construction interaction and requires high-precision perception, including the working surface, robotic arm, bucket, construction materials, obstacles, etc.

[0028] A continuous 3D point cloud model refers to a set of 3D spatial points that are continuous in the time dimension, dense in the spatial dimension, have high geometric accuracy, and have a complete structure. Each point contains information such as 3D coordinates, color, and confidence level, and can accurately reproduce the 3D geometric structure and texture features of the near-field environment of the operation in real time.

[0029] In some embodiments, based on the fusion processing of the binocular vision data and the dToF depth data, a continuous three-dimensional point cloud model of the near-field environment of the operation and the corresponding point cloud fusion confidence are obtained, including the following sub-steps S1-1-S1-5: S1-1. Correct and estimate the disparity of the binocular vision data to generate a binocular depth map, and determine the corresponding binocular confidence based on left-right consistency, matching cost, texture intensity, edge stability and temporal stability.

[0030] The binocular vision data consists of binocular images captured by a binocular camera. Specifically, the binocular images undergo distortion correction, stereo correction, and epipolar alignment to eliminate lens distortion and align corresponding pixels along the same epipolar line. A stereo matching algorithm is used to calculate the disparity value of corresponding pixels in the binocular images, which, combined with the baseline distance within the binocular camera, is used to generate a binocular depth map. Simultaneously, five types of features are extracted pixel-by-pixel: left-right consistency, matching cost, texture intensity, edge stability, and temporal stability. Based on these features, the corresponding binocular confidence level is determined. Left-right consistency refers to the degree of agreement between the two-way matching results of the left and right images; the smaller the deviation, the higher the confidence level. Matching cost is the difference in features between the left and right pixels; a smaller difference indicates a more reliable matching result. Texture intensity refers to the richness of texture in a local area of ​​the image; richer texture leads to more stable stereo matching results. Edge stability refers to the matching stability of pixels in the scene's edge areas; clear edge features result in higher reliability of the depth result. Temporal stability: The degree of continuous change in depth results at the same location in adjacent frames. The smaller the depth fluctuation in a short period of time, the higher the confidence level.

[0031] S1-2. The dToF depth data is denoised and multipath suppressed to generate a dToF depth map, and the corresponding dToF confidence level is determined based on echo intensity, signal-to-noise ratio, multipath risk, flying point probability, neighborhood depth consistency and historical stability.

[0032] In practice, the dToF depth data is filtered and denoised to remove random noise and environmental interference; multipath reflection interference is identified and suppressed to correct depth deviation; and a stable dToF depth map is generated after removing invalid measurements. Simultaneously, echo intensity, signal-to-noise ratio (SNR), multipath risk, flying point probability, neighborhood depth consistency, and historical stability features are extracted pixel by pixel. The corresponding dToF confidence level is determined based on these features. Specifically, echo intensity refers to the intensity of the reflected light signal received by the sensor; a stronger signal indicates a more reliable ranging result. Signal-to-noise ratio (SNR) is the ratio of effective signal to interference noise; a higher ratio indicates better data quality. Multipath risk is the probability of multiple reflections of light; a higher risk indicates a greater depth deviation. Flying point probability is the probability of abnormal ranging points that significantly deviate from the true distance. Neighborhood depth consistency refers to the degree of fit between the current pixel and the depth values ​​of surrounding pixels; a smoother local depth transition indicates higher confidence.

[0033] S1-3. Based on the deep learning model, and combining the scene and data feature parameters, as well as the stereo confidence and the dToF confidence, determine the pixel-wise fusion weights corresponding to the stereo depth map and the dToF depth map respectively.

[0034] In practice, the binocular depth map, dToF depth map, scene feature parameters (including at least one of texture, reflectivity, distance, viewpoint change, dust / occlusion, and historical stability), pixel-by-pixel binocular confidence score, and pixel-by-pixel dToF confidence score are input into the trained deep learning fusion model. The model extracts multi-scale features and learns the correlation between scene features, data features, and confidence scores. It assigns binocular fusion weights and dToF fusion weights pixel by pixel, with the sum of the weights being 1. The binocular weights are increased in regions with rich details and high binocular confidence, while the dToF weights are increased in regions with weak texture, dust, occlusion, and high dToF confidence. The fused depth data is obtained by weighted summation by pixels.

[0035] S1-4. Based on the pixel-wise fusion weights of the stereo depth map and the dToF depth map, perform adaptive weighted fusion processing on the stereo depth map and the dToF depth map to obtain fused depth data and determine the corresponding point cloud fusion confidence.

[0036] Based on the one-to-one correspondence of pixels, the fusion weights of the stereo and dToF obtained in step three are called; weighted operations are performed on the stereo depth value and dToF depth value at the same pixel position to obtain fused depth data; the fusion weight of the current pixel is reused to perform weighted calculations on the stereo confidence and dToF confidence at that position to obtain pixel-level point cloud fusion confidence.

[0037] S1-5. The fused depth data is combined with the point cloud fusion confidence score to perform voxel filtering and outlier removal processing to generate a continuous three-dimensional point cloud model of the near-field environment of the operation.

[0038] In practice, based on the fused depth data, binocular camera intrinsic parameters, extrinsic parameters, and timestamps, each pixel is back-projected into three-dimensional space to generate a single-frame dense three-dimensional point cloud. Using the point cloud fusion confidence level as the criterion, low-confidence three-dimensional points are downweighted or preferentially removed, while high-confidence valid three-dimensional points are retained. Voxel filtering is used to downsample and homogenize the point cloud, eliminating redundant points. Filtering methods are used to remove outliers caused by dust, reflection, and noise. Temporal correlation, registration, and fusion are performed on continuous frame point clouds to remove dynamic interference points, generating a continuous three-dimensional point cloud model of the near-field working environment that is temporally continuous, spatially dense, geometrically accurate, and structurally complete.

[0039] For example, see Figure 1c The diagram illustrates the depth data fusion process. The specific fusion process includes: after distortion correction and epipolar correction of the original binocular images, stereo disparity estimation is performed, and the original binocular depth map is obtained from the disparity. Based on left-right matching consistency, matching cost, texture, edge, and temporal stability, the binocular confidence is solved pixel-by-pixel. This characterizes the reliability of the depth of each pixel in the binocular system. The dToF sensor outputs raw time-of-flight depth data. First, noise filtering and multipath reflection suppression are performed to eliminate ranging flypoints and multipath errors, resulting in an optimized dToF depth map. Then, based on echo intensity, signal-to-noise ratio, multipath risk, neighborhood depth consistency, and inter-frame historical stability, dToF confidence is generated pixel-by-pixel. This includes factors such as distance, texture, object reflectivity, shooting angle, temporal history stability, and binocular confidence. dToF confidence level All data is fed into a deep learning network, where the model autonomously outputs pixel-by-pixel dynamically fused weights. (Binocular weight) (dToF weights); The fusion formula is adopted: ; The two depth maps are adaptively weighted using dynamic weights to generate the final fused depth map. .

[0040] It should be noted that, generally speaking, for the 0-3m near-field range: binocular detail accuracy is superior, and the model automatically increases the binocular weights. , suppress Prioritize the use of binocular depth; in the 2-5m mid-range: binocular and dToF performance are balanced, with smooth and continuous weight transitions, and the two depth channels are fused collaboratively; in the 5-50m long-range: dToF ranging has stronger stability, and the model is elevated. ,reduce The dToF depth is used as the main data; the weights are automatically allocated by the network throughout the process, without manual switching of thresholds.

[0041] Fusion depth By combining camera intrinsic back projection, a near-field raw point cloud is generated; based on the point cloud fusion confidence obtained above, voxel filtering and outlier removal are performed to filter out low-confidence outliers and reduce the point cloud density; ground / work surface candidate region segmentation is performed on the point cloud to separate the work surface geometric model for subsequent spatial distance and deviation calculation.

[0042] Taking excavator pit excavation as an example, step S1 is as follows: For pit excavation operations characterized by high dust levels, weak texture, and severe near-field occlusion, the binocular vision data undergoes distortion correction, stereo correction, and disparity estimation to generate a near-field binocular depth map and binocular confidence score. The binocular depth map clearly restores details in textured areas such as the pit edge and bucket outline. The dToF depth data undergoes denoising, multipath suppression, and invalid value removal to generate a stable dToF depth map and dToF confidence score. The dToF depth map accurately outputs depth information for the pit interior and dust-occluded areas. The two types of depth maps, confidence scores, and scene dust concentration and texture density parameters are input into a deep learning fusion model. The model automatically assigns fusion weights: at the pit edge... In areas with rich textures, such as the bucket surface, the binocular camera is weighted at 80% and the dToF camera at 20% to preserve details. In areas inside the pit and obscured by dust, the binocular camera is weighted at 20% and the dToF camera at 80% to ensure stability. A high-precision fused depth map is generated. Based on the back projection of the fused depth map, a single-frame 3D point cloud of the pit is generated. The continuous frame point cloud is processed by temporal correlation, voxel filtering, and outlier removal to remove dust noise points and dynamic interference points. Finally, a continuous 3D point cloud model of the pit's near-field environment is constructed that is temporally continuous, spatially dense, and has stable accuracy. This model accurately restores the 3D geometric structure of the pit's inner wall, bucket, and working face, providing high-precision 3D data support for subsequent pit working face modeling, bucket feature extraction, and spatial deviation calculation.

[0043] The beneficial effects of this step are as follows: by using deep learning to adaptively fuse binocular vision data and dToF depth data, the limitations of a single perceptual mode are overcome, while taking into account both texture details and stability under harsh working conditions; after filtering and optimization, a high-precision continuous 3D point cloud is generated, which accurately restores the near-field 3D structure of the operation, providing a reliable 3D data foundation for subsequent operation surface modeling, end feature extraction and spatial relationship calculation.

[0044] In step S2, joint optimization processing refers to using binocular vision, dToF depth, and IMU inertial as joint constraints, and through a multi-sensor fusion optimization algorithm, performing global and local joint calculations of the equipment pose (or the pose of the construction machinery body) to achieve continuous, high-precision, and stable pose output, avoiding single-sensor drift or error accumulation. The construction machinery body refers to the fixed structural parts of the construction machinery, such as the body, frame, and cab, as opposed to movable end-effectors (such as robotic arms and buckets).

[0045] Continuous pose of equipment refers to the six-degree-of-freedom state of the engineering machinery body that changes continuously over time in the world coordinate system, including three-dimensional position (X, Y, Z coordinates) and three-dimensional attitude (pitch angle, roll angle, yaw angle). Each moment corresponds to a set of pose data, forming a continuous pose sequence.

[0046] The core of step S2 is: While constructing a continuous 3D point cloud model of the near-field operation in parallel, simultaneously fuse binocular vision, dToF depth, and inertial measurement. Through feature matching, inter-frame registration, inertial pre-integration, and multi-sensor joint optimization algorithms, the six-degree-of-freedom continuous pose of the engineering machinery body in the world coordinate system is calculated in real-time, continuously, and with high precision. This provides a unified and stable pose reference for subsequent end-effector state calculation, data coordinate transformation, and scene model construction. Additionally, if global positioning data exists, it can be fused to provide global constraint calibration. Specifically, when fusing data from step 4 to determine the continuous pose of the equipment, steps S2-1, S2-2, ..., S2-5 are included: S2-1. Extract image features and depth features from binocular vision data and dToF depth data.

[0047] In the parallel process of constructing a continuous 3D point cloud model, the binocular images contained in the binocular vision data are preprocessed to remove distortion interference and extract image features that can characterize scene structure and texture information, including stable features such as corner points, edges, texture blocks, and key points. At the same time, depth features that can characterize spatial geometry are extracted from the depth information corresponding to the dToF depth data, including geometric features such as planes, curved surface edges, depth abrupt boundaries, and spatial corner points. The extracted image features and depth features are correlated and matched to form an image-depth joint feature set, which provides a stable feature basis for subsequent inter-frame pose estimation.

[0048] S2-2. Based on image features and depth features, estimate the relative pose change between adjacent frames through inter-frame registration, iterative nearest point matching, or feature matching.

[0049] Specifically, based on the time-series continuous image-depth joint feature set, at least one of the following methods is used: inter-frame registration, iterative nearest point matching, or feature matching, to match the correspondence of joint features between two adjacent frames; by minimizing the feature matching error between adjacent frames, the relative rotation matrix and relative translation vector between the two frames are solved, thereby estimating the relative pose change between adjacent frames and characterizing the spatial position and attitude change of the engineering machinery body at adjacent sampling times.

[0050] In the actual engineering implementation of inter-frame registration, iterative nearest point matching, or feature matching, the point cloud fusion confidence obtained above is used as the registration weight. Weighted constraints are applied to unfavorable working conditions commonly found in construction sites, such as dust interference, target surface reflection, dynamic objects in the scene, and low-confidence invalid depth points. The registration weights of feature points corresponding to dust areas, reflective areas, and dynamic objects, as well as low-confidence depth points, are reduced to decrease the influence of the above invalid features in the inter-frame matching solution, thereby improving the robustness of pose solution in complex construction environments.

[0051] S2-3. Perform pre-integration processing on the inertial measurement data to generate incremental motion constraint data between adjacent frames.

[0052] The high-frequency acceleration and angular velocity information contained in the inertial measurement data are pre-integrated to compensate for the zero bias error and noise interference of the inertial measurement unit. Within the time interval between two adjacent frames, the acceleration and angular velocity are integrated to calculate the position increment, velocity increment and attitude increment between adjacent frames, generating incremental motion constraint data. This data is used to characterize the motion trend of the engineering machinery body between adjacent frames and provide continuous motion constraints for pose calculation.

[0053] S2-4. When the global positioning data is valid, use the global coordinate information it provides as a constraint for pose estimation.

[0054] The system monitors the signal status and validity of global positioning data in real time. When the global positioning data is valid and the signal quality meets the preset accuracy requirements, it extracts the three-dimensional global coordinate information of the engineering machinery body in the world coordinate system provided by the global positioning data. This global coordinate information is used as an absolute position constraint term and introduced into the pose estimation process to correct the cumulative error generated during the local pose calculation process, ensuring that the pose result is aligned with the world coordinate system.

[0055] S2-5. When global positioning data is degraded or invalid, reduce its constraint strength or stop using the constraint term.

[0056] Specifically, the constraint weights corresponding to global localization can be linearly or piecewise reduced based on the severity of degradation: the more severe the degradation, the smaller the weight value, weakening the constraint effect of global coordinates on pose calculation; in this case, the optimization still retains the global localization term, but the influence of this constraint in the overall solution is compressed. When the data is invalid, the constraint weights corresponding to global localization are reset to zero, and global coordinate constraints are no longer included in this optimization.

[0057] S2-6. Using sliding window optimization, extended Kalman filtering, or factor graph optimization methods, the relative pose change, incremental motion constraint data, and pose estimation constraint terms are jointly estimated to determine the continuous pose of the device.

[0058] Specifically, at least one joint optimization method among sliding window optimization, extended Kalman filtering, or factor graph optimization is selected to construct a joint optimization objective function that includes relative pose change, incremental motion constraint data, and global coordinate constraint terms. Through nonlinear iterative solution, the constraint relationships of multi-source data are integrated to minimize the overall estimation error. Finally, the three-dimensional position and three-dimensional attitude of the engineering machinery body in the world coordinate system that change continuously over time are obtained, forming a continuous, stable, and high-precision continuous pose sequence.

[0059] For example, taking excavator pit excavation as an example, step S2 specifically involves: simultaneously constructing a continuous 3D point cloud model of the pit near field, and simultaneously performing continuous pose calculation of the excavator body: extracting stable key feature points such as pit edges and ground markers from binocular images, and extracting depth features such as pit planes and bucket edges from dToF depth data to form an image-depth joint feature set; matching and ICP registration of the continuous frame feature sets to estimate the relative position offset and attitude rotation of the excavator body between adjacent frames; pre-integrating the IMU high-frequency inertial measurement data to generate inter-frame inertial increment constraint data, and constraining the position... Attitude change and drift suppression: When RTK global positioning data is valid, the centimeter-level coordinates and heading angles output in the engineering coordinate system are used as global constraints to calibrate local pose; a factor graph optimization algorithm is adopted to integrate inter-frame relative pose, inertial constraints, and RTK global constraints to optimize and solve the three-dimensional position (X, Y, Z) and pitch, roll, and heading angles of the excavator body in the world coordinate system in real time, and output a pose sequence that changes continuously over time, accurately reflecting the real-time position and attitude of the excavator body, providing a stable and high-precision pose reference for subsequent bucket state calculation, pit model coordinate transformation, and spatial deviation calculation.

[0060] The beneficial effects of this step are as follows: Through multi-sensor joint optimization using binocular vision, dToF depth, IMU inertial, and RTK global positioning, a deep fusion of local pose tracking and global coordinate calibration is achieved, effectively solving the inherent defects of single-sensor pose calculation: pure visual pose is easily drifted due to changes in scene texture and lighting; pure inertial pose error accumulates rapidly over time; pure RTK can only provide overall position and cannot output high-frequency continuous posture; joint optimization can complement the advantages of various sensors, suppress error accumulation, and improve the accuracy, stability, continuity, and robustness of pose calculation; the output continuous pose of the body provides a unified, stable, and high-precision pose reference for subsequent end-effector state calculation, unified conversion of multi-source data coordinates, and construction of multi-layer scene semantic models, ensuring the accuracy and reliability of subsequent spatial relationship calculations and guidance command generation, and adapting to working conditions such as strong vibration of engineering machinery, complex outdoor environments, and RTK signal fluctuations.

[0061] In step 102, the end-of-work mechanism refers to the movable end component of the construction machinery used to directly perform construction operations, including at least one of the following: excavator boom, stick, bucket, bucket teeth, bucket tip, and bucket edge.

[0062] The end-effector features refer to the key features extracted from continuous 3D point cloud models or binocular vision data that characterize the geometry and spatial position of the end-effector, including key points of the mechanism (such as bucket blade tip, joint axis center), rod edges, end profile, planes, corners, etc.

[0063] Joint angles refer to the rotation angles of various joints in the end effector of construction machinery, such as the angle between the excavator boom and the frame, the angle between the stick and the boom, and the angle between the bucket and the stick. They are core parameters characterizing the attitude of the end effector.

[0064] Key point coordinates refer to the three-dimensional coordinates of key feature points of the end-of-operation mechanism in the world coordinate system, such as the three-dimensional position of the bucket tip, bucket edge point, and joint center point, which directly reflects the spatial position of the end-of-operation point.

[0065] The envelope model refers to the three-dimensional spatial contour model of the end effector, which is usually a simplified three-dimensional geometry (such as a cube, prism, or mesh model). It is used to characterize the overall spatial occupancy of the end effector and is an important basis for judging collision risk and safe distance.

[0066] The core of step S102 is to construct a kinematic model of the end effector based on the prior geometric parameters of the engineering machinery equipment, extract key geometric features of the end effector from the continuous three-dimensional point cloud model or binocular vision data, solve the joint angles through feature matching and attitude estimation algorithms, and combine the continuous pose of the body with the kinematic model to solve the state information of the key points, edges and envelope of the end effector in the world coordinate system, so as to realize the real-time and accurate state perception of the end effector.

[0067] In practice, based on the prior geometric parameters of the equipment and the features of the end-effector extracted from continuous 3D point cloud models or binocular vision data, the state information of the end-effector of the construction machinery is calculated, including S1021-S1024: S1021. Based on the prior geometric parameters of the equipment, construct the linkage kinematic model of the engineering machinery.

[0068] Specifically, based on the prior geometric parameters of the construction machinery calibrated and stored before leaving the factory, including the length of each link such as the boom and stick, the distance between joint axes, the coordinates of the joint rotation center, the connection relationship of the links, the range of joint rotation, and the transformation matrix between each joint coordinate system and the base coordinate system; according to the kinematic principle of the robotic arm linkage mechanism, a linkage kinematic model from the construction machinery body base to the end-effector mechanism is established, clarifying the direction of each joint rotation axis, the spatial constraint relationship between links, and the positive kinematic mapping relationship, that is, the mathematical relationship of deriving the end-effector pose from the joint angle, providing a theoretical model basis for the subsequent end-effector state calculation.

[0069] S1022. Extract key geometric features of the end-effector from continuous 3D point cloud models or binocular vision data. Key geometric features include key points of the mechanism, edges of rods, or end contours.

[0070] Specifically, from the completed continuous 3D point cloud model, point cloud data corresponding to the end-of-life mechanism is separated through point cloud segmentation, cluster analysis, and edge detection algorithms. Key points of the mechanism, such as boom edge, stick edge, bucket backplate contour, bucket tooth tip, bucket blade tip, bucket side edges, and joint axis center, as well as rod contour lines, bucket edge lines, and the overall outer boundary shape of the bucket, are extracted. Alternatively, from binocular vision data (binocular images), the end-of-life mechanism region is identified through image segmentation, edge extraction, and corner detection algorithms. Two-dimensional key points, edge lines, and contour lines are extracted, and then combined with depth information back projection to obtain three-dimensional key geometric features, forming a feature set for attitude calculation.

[0071] S1023. If the construction machinery is equipped with joint sensors, CAN bus data, or hydraulic cylinder stroke data, then such data shall be used as prior information or constraints; if the above data is not available, then the joint angle of the end-effector shall be estimated based on key geometric features by key point detection, edge extraction, or geometric matching.

[0072] When the equipment is equipped with joint sensors, can read CAN bus data, or has a hydraulic cylinder stroke acquisition component, the above-mentioned measured working condition data is extracted. The theoretical joint angle is calculated based on the factory-calibrated linkage geometry relationship. This theoretical angle is used as prior information and boundary constraint for solving the joint angle. In the process of solving the joint angle based on image and point cloud feature geometric matching, this constraint is used to narrow the matching search range and verify the rationality of the solution result. In the case of missing visual features, the measured converted angle can be directly used as the output. When the whole machine is not equipped with the above-mentioned sensing components, the extracted key geometric features of the end-effector are compared with the preset theoretical feature template in the linkage kinematic model. By matching the theoretical key point position through key point detection, fitting the theoretical link edge direction through edge extraction, and aligning the theoretical end-effector contour shape through geometric matching, the correspondence between the actual features and the theoretical model is established. The least squares method or iterative optimization algorithm is used to minimize the spatial error between the actual features and the theoretical features, and the real-time rotation angle of each joint is calculated to obtain the joint angle sequence of the end-effector.

[0073] S1024. Based on the linkage kinematics model, joint angles, and continuous pose of the equipment, solve the pose of the key points, end edges, and end envelope of the end-effector in the world coordinate system, which serves as the state information of the end-effector.

[0074] Specifically, the linkage kinematic model constructed in S1021, the joint angles calculated in S1023, and the determined continuous pose of the equipment are substituted into the forward kinematic equations. By sequentially solving the coordinate transformation matrices from each joint coordinate system to the equipment coordinate system and then to the world coordinate system, the three-dimensional coordinates and spatial poses of each key point and end edge of the end-effector in the world coordinate system are derived and calculated in the forward direction. Combined with the external dimensions of the end-effector, the pose of the end-effector envelope model adapted to its spatial shape in the world coordinate system is generated. The above information is integrated to form the state information of the end-effector, which includes joint angles, key point coordinates, end edge poses, and end-effector envelope poses.

[0075] For example, taking excavator pit excavation as an example, step S102 is as follows: Based on the prior geometric parameters of the excavator equipment (boom length, stick length, bucket size, joint spacing), combined with the prior constraints such as the equipment CAD whole machine model, mechanism outline, component color, and linkage motion continuity, a kinematic model of the boom-stick-bucket linkage is constructed; relying on the above-mentioned multiple prior constraints as screening criteria, and in conjunction with point cloud segmentation and edge detection algorithms, the bucket point cloud is separated from the near-field continuous 3D point cloud model of the pit, and 3D key features such as the bucket tip, bucket edge, and stick joint center are extracted; the actual extracted bucket features are matched with the theoretical features of the kinematic model, and the error is minimized through iterative optimization algorithms to solve for the boom joint angle, stick joint angle, and bucket joint angle. Angles; combining the linkage kinematics model, real-time joint angles, and continuous pose of the excavator body, the three-dimensional coordinates of the bucket tip, bucket edge points, and joint center points in the world coordinate system are solved in a forward manner; based on the theoretical dimensions of the bucket and the point cloud contour, a three-dimensional mesh envelope model of the bucket is constructed; finally, the joint angles, key point coordinates, and envelope model are integrated to form complete bucket state information, accurately reflecting the real-time attitude of the bucket, the spatial position of the working point, and the overall spatial contour, providing core data support for subsequent calculations of the spatial relationship between the bucket and the foundation pit design surface, the working surface, and the underground pipeline danger zone.

[0076] The beneficial effects of step S102 are as follows: by constructing a linkage kinematic model, extracting key geometric features of the end effector, inversely solving the joint angles, and combining them with the body pose for forward solving, real-time, accurate, and complete perception of the state of the end effector mechanism is achieved; the problem that existing technologies cannot directly obtain the end effector joint angles, key spatial points, and envelope range is solved, providing core quantitative basis for subsequent calculation of the spatial relationship between the end effector mechanism and the working surface, construction design information, and safety constraint information, ensuring the accuracy and effectiveness of operation guidance instructions and safety warning information, and improving construction precision and operation safety.

[0077] In step S103, the safety constraint information includes at least one of underground pipelines, high-voltage lines, slope danger zones, personnel restricted areas, equipment restricted areas, and dynamic safety boundaries; the working surface geometric model refers to a geometric model constructed based on a continuous three-dimensional point cloud model that represents the three-dimensional shape of the actual working surface of the engineering machinery, including information such as working surface elevation, slope, flatness, contour boundary, and curved surface shape, which accurately reflects the actual construction surface condition on site.

[0078] The world coordinate system refers to a unified global engineering coordinate system, usually a geodetic coordinate system or engineering coordinate system preset for the construction site. It serves as a unified spatial reference for all entities on site (engineering machinery, work surfaces, design surfaces, and hazardous areas), ensuring the consistency of spatial location of multi-source data.

[0079] Unified registration refers to mapping data (construction design information, safety constraint information) from different sources and coordinate systems to the same world coordinate system through coordinate transformation, point cloud registration, feature alignment, and other methods, thereby achieving spatial alignment and fusion of multi-source data.

[0080] A multi-layered operational scenario semantic model refers to a hierarchical three-dimensional scenario model that integrates the actual on-site state, equipment state, construction benchmarks, and safety constraints, divided according to semantic levels. It includes an equipment layer, an operational surface layer, a design surface layer, and a hazardous area layer. Each layer corresponds to a type of semantic information, realizing the structured and semantic organization of scenario information.

[0081] The core of this step is to: construct the actual working surface geometric model based on the continuous 3D point cloud model; transform the end mechanism status information and working surface geometric model to the world coordinate system based on the continuous pose of the equipment; map the construction design information and safety constraint information to the same world coordinate system through coordinate transformation and feature registration; organize the data in layers according to the semantic hierarchy of equipment, working surface, design surface and danger zone, construct a multi-layer working scene semantic model, and realize the spatial unification and semantic structured fusion of multi-source data.

[0082] In practice, step S103 includes S1031-S1034: S1031. From the continuous three-dimensional point cloud model, segment and extract the target area point cloud corresponding to the construction scene.

[0083] For the completed continuous 3D point cloud model of the near-field environment of the operation, based on the scope of the construction scene, the boundary of the operation area and the activity range of the end mechanism of the construction machinery, algorithms such as point cloud segmentation, region growth, distance threshold screening and planar clustering are used to remove non-construction scene point clouds such as the construction machinery body, the robotic arm, invalid noise points and distant background points from the overall point cloud; the target area point cloud directly corresponding to the current construction scene is accurately separated and extracted. The target area point cloud mainly includes the 3D point cloud data of the working surface, the construction soil, the area to be excavated or the area to be leveled.

[0084] S1032. Fit or mesh the point cloud of the target area to generate a geometric model of the working surface that matches the construction scene.

[0085] Specifically, the extracted point cloud of the target area is fitted using methods such as surface fitting, plane fitting, polynomial fitting, and least squares fitting, or meshed using methods such as triangular meshing, polygonal meshing, and regular meshing. Through point cloud denoising, smoothing, hole filling, and boundary optimization, a geometric model that can accurately represent the three-dimensional morphology of the actual working surface in the current construction scene is constructed. The geometric model of the working surface includes geometric information such as elevation distribution, slope, flatness, boundary contour, and surface shape, which completely matches the actual state of the current construction scene.

[0086] S1033. Based on the continuous pose of the equipment, the status information of the end-of-operation mechanism and the geometric model of the working surface are converted to the world coordinate system, and the construction design information and safety constraint information are uniformly registered to the world coordinate system.

[0087] Specifically, based on the defined continuous pose of the equipment, a real-time coordinate transformation matrix is ​​established between the equipment coordinate system and the world coordinate system. Through coordinate transformation operations, the calculated end-effector state information and the geometric model of the working surface generated by S are all transformed from the equipment coordinate system to the world coordinate system. At the same time, the pre-stored construction design information and safety constraint information are extracted. Based on the control point and feature point matching or point cloud registration algorithm in the world coordinate system, the construction design information and safety constraint information are uniformly registered to the world coordinate system through coordinate translation, rotation, and scaling adjustments, thereby achieving the unification of the spatial reference of multi-source data.

[0088] S1034. The unified mapped continuous pose of the equipment and the status information of the end-effector mechanism are incorporated into the equipment layer, the geometric model of the working surface is incorporated into the working surface layer, the construction design information is incorporated into the design surface layer, and the safety constraint information is incorporated into the hazardous area layer to construct a multi-layer semantic model of the working scenario.

[0089] The multi-source data, uniformly mapped to the world coordinate system, is organized hierarchically according to semantic hierarchy. Continuous device pose and end-effector status information are grouped into the device layer to represent the real-time working status of construction machinery. The geometric model of the working surface is grouped into the working surface layer to represent the actual surface morphology of the current construction scenario. Construction design information is grouped into the design surface layer to represent preset construction benchmarks and target requirements. Safety constraint information is grouped into the hazardous area layer to represent construction restricted areas and safety boundaries. By integrating data from each level, a structured, semantic, and hierarchical multi-layered semantic model of the operational scenario is constructed, enabling hierarchical management, unified access, and semantic association of construction scenario information.

[0090] Taking excavator pit excavation as an example, the implementation process of step S103 includes: separating the pit working face point cloud from the near-field continuous three-dimensional point cloud model of the pit through ground filtering and plane fitting algorithms, and removing non-working face point clouds such as excavator and slag; performing meshing and surface reconstruction on the working face point cloud to generate a geometric model of the pit working face containing the pit inner wall elevation, slope, flatness, and excavation outline, accurately restoring the actual excavation surface morphology of the pit; establishing a transformation matrix between the equipment coordinate system and the engineering world coordinate system based on the continuous pose of the excavator body, transforming the bucket key point coordinates, envelope model, and pit working face geometric model to the world coordinate system; extracting construction design information such as the three-dimensional model of the pit design surface, the target excavation outline, and the design elevation, and extracting ground... Safety constraint information such as the scope of underground pipelines and the dangerous boundaries of the foundation pit slope is collected. Based on the site control point and point cloud registration algorithm, the design information and safety constraint information are uniformly registered to the engineering world coordinate system, so that the designed foundation pit surface, dangerous area boundary and the actual foundation pit working surface and bucket are on the same spatial reference. A multi-layer semantic model of the operation scene is constructed according to semantic layering: the equipment layer contains the excavator body pose and bucket status information, the working surface layer contains the actual foundation pit working surface model, the design surface layer contains the registered foundation pit design information, and the dangerous area layer contains the registered pipeline and slope dangerous area information. This realizes the spatial unification and semantic structured fusion of multi-source data of the foundation pit scene, and provides complete scene data support for subsequent calculation of the spatial relationship parameters of the bucket, foundation pit working surface and design and dangerous area.

[0091] The beneficial effects of step S103 are as follows: By extracting point clouds of the target area, constructing geometric models of the work surface, unifying the registration of coordinates of multi-source data, and constructing multi-layer semantic models, spatial alignment and semantic fusion of the actual state of the construction scene, the state of the engineering machinery, the construction design benchmark, and the safety constraint information under the same world coordinate system are achieved; the problem of inconsistent benchmarks of on-site data, design data, and safety data, and the inability to directly correlate and analyze them is effectively solved; a complete, unified, and semantically clear scene data foundation is provided for subsequent accurate calculation of the spatial relationship parameters between the end mechanism of the operation, the work surface, and the construction design information and safety constraint information; the accuracy, comprehensiveness, and efficiency of spatial relationship quantitative calculation are ensured; the barriers between perception, design, and safety data are broken down; and a complete digital expression of the construction scene is formed.

[0092] In step S104, the end effector refers to the movable end component of the construction machinery used to directly perform construction operations (such as the excavator bucket, boom, and stick), and its status information includes joint angles, key point coordinates, and end effector envelope. The working surface refers to the actual working surface directly acted upon by the construction machinery in the current construction scenario, accurately represented by the geometric model of the working surface generated in S103, including geometric information such as elevation, slope, and contour. Spatial relationship parameters are numerical indicators that quantitatively represent the geometric relationships between the end effector, the working surface, construction design information, and safety constraint information. They are the core quantitative link connecting the scenario model with guidance instructions and early warning information, including distance, deviation, attitude difference, spatial intrusion status, and alignment degree.

[0093] The core of step S104 is: based on the data aligned at each level in the semantic model of the multi-layer operation scenario, through algorithms such as three-dimensional spatial geometric calculation, distance measurement, deviation analysis, and collision detection, to quantify and calculate various spatial relationship parameters between the end mechanism of the operation, the actual working surface and the construction design information and safety constraint information, respectively, to achieve accurate quantitative representation of the geometric relationship between the above objects, and to provide quantitative basis for subsequent operation guidance and safety early warning.

[0094] In specific implementation, step S104 includes the following steps S1041-S1044: S1041. Retrieve data corresponding to the equipment layer, work surface layer, design surface layer, and hazardous area layer in the multi-layer operation scenario semantic model.

[0095] The system invokes a pre-built multi-layered semantic model of the operational scenario. All data in this model is uniformly mapped to the world coordinate system, eliminating the need for additional coordinate transformation. It accurately retrieves core data corresponding to each level of the model. Equipment layer data includes continuous equipment pose and end-effector status information (including key point coordinates, joint angles, and end-effector envelope model). Operational surface layer data is the geometric model of the operational surface generated by S105 (including elevation, slope, and contour information). Design surface layer data consists of pre-stored construction design information (including 3D model of the design surface, target construction contour, preset operational path, and design elevation). Hazardous area layer data contains pre-stored safety constraint information (including hazardous area boundaries, underground pipeline range, and slope hazard boundaries). The retrieved data undergoes format verification and validity filtering to eliminate invalid data, ensuring the accuracy of subsequent calculations.

[0096] S1042. Based on the data corresponding to the equipment layer and the design surface layer, determine the deviation of the normal distance between the key points of the end-of-operation mechanism and the design surface in the construction design information, as well as the spatial position deviation and attitude deviation between the current position of the end-of-operation mechanism and the target construction outline and preset operation path in the construction design information.

[0097] Extract the coordinates of key points (such as bucket tip, joint center point, etc.) of the end-effector in the equipment layer; extract the 3D model of the design surface, target construction outline, and preset operation path data of the construction design information in the design surface layer; through 3D spatial geometric calculations, solve the normal distance from each key point of the end-effector to the design surface, compare it with the preset design distance to obtain the normal distance deviation, and clarify the degree of deviation between the key points and the design surface; for example, for the normal distance from the key point to the design surface, rely on the plane equation The calculation involves finding the normal vector of the design surface (n), the plane constant of the design surface (b), and the spatial coordinates of the key point of the bucket tip (P). The calculated d is the measured normal distance of the tip relative to the design surface. The difference between the measured distance and the preset design distance is used to obtain the normal distance deviation. A spatial coordinate comparison algorithm is used to calculate the spatial position deviation between the current position coordinates of the end-effector and the boundary coordinates of the target construction contour, as well as the coordinates of the preset work path, to determine the direction and amount of deviation. For example, this step uses the deviation calculation formula... Complete the deviation calculation: In the formula This represents the current actual position coordinates of the end effector. The coordinates of the target construction outline and the corresponding positional coordinates of the preset work path are used to calculate the overall deviation vector by performing a vector difference operation between the two; according to the formula... The deviation vector is decomposed into four-dimensional components, with each of the four parameters corresponding to one of the four types of physical deviations: For normal / lateral position deviation, For longitudinal deviation along the preset path, For elevation (height) deviation, The attitude orientation deviation is determined by using the positive and negative signs of four sub-deviations to identify the direction of deviation and the magnitude of the deviation to fully quantify the spatial offset of the end effector relative to the design profile and the operation path. By comparing attitude angles, the attitude deviation between the current attitude of the end effector and the design preset attitude is calculated, covering angle deviations such as pitch angle and roll angle, fully characterizing the fit between the end effector and the construction design requirements.

[0098] S1043. Based on the data corresponding to the equipment layer and the working surface layer, determine the minimum distance and contact state between the working end mechanism envelope and the working surface, which is constructed from the key points, edge points and simplified geometric model of the working end mechanism; the type of the working end mechanism envelope includes at least one of convex hull, bounding box, capsule body or grid envelope.

[0099] Extract the end-effector envelope model from the equipment layer, and extract the 3D coordinates and contour information of the working surface geometry model from the working surface layer. Using a collision detection and distance calculation algorithm, traverse all surface nodes of the end-effector envelope and the surface nodes of the working surface, calculating the shortest straight-line distance between them, which is the minimum distance between the end-effector envelope and the working surface. Specifically, the minimum distance can be calculated using the following formula. : , For all discrete sampling points on the surface of the envelope, The sampling points are used for the point cloud of the working face. The contact status is determined based on the minimum distance value. If the minimum distance is greater than 0, it is determined to be a non-contact state. If the minimum distance is equal to 0, it is determined to be a contact state. If the minimum distance is less than 0, it is determined to be an excessive contact (such as the bucket being excessively embedded in the working face). The coordinate information of the contact point is recorded simultaneously to provide a basis for subsequent operation guidance.

[0100] S1044. Based on the data corresponding to the equipment layer and the hazardous area layer, determine the safe distance and spatial intrusion status between the engineering machinery body, the end-of-line mechanism and the hazardous area defined by the safety constraint information.

[0101] Extract the continuous pose (3D position and attitude) of the equipment in the equipment layer and the end-effector envelope model of the end-effector mechanism. Extract the 3D model and boundary coordinates of the hazardous area defined by the safety constraint information in the hazardous area layer. Calculate the minimum safe distance from the center point of the engineering machinery body to the boundary of the hazardous area, and the minimum safe distance from the end-effector envelope to the boundary of the hazardous area, using a 3D spatial distance calculation algorithm. Compare the calculated minimum safe distance with a preset safe distance threshold to determine the spatial intrusion status: if the minimum safe distance is greater than the threshold, it is determined to be in an intrusion-free state; if the minimum safe distance is less than or equal to the threshold but greater than 0, it is determined to be an imminent intrusion state; if the minimum safe distance is less than 0, it is determined to be an intrusion state. Simultaneously, calculate the intrusion depth and intrusion range to clarify the degree of safety risk and provide quantitative support for subsequent safety early warning.

[0102] The beneficial effect of step S104 is that it transforms the spatial relationships of objects in the multi-layer operation scenario semantic model from qualitative descriptions into quantitative parameters, effectively solving the problem in the existing technology that it is impossible to accurately quantify the geometric relationships between the end mechanism, the working surface and the design target and safety boundary; the quantified spatial relationship parameters provide a clear and reliable quantitative basis for the accurate generation of subsequent operation guidance instructions and the hierarchical output of safety warning information, ensuring the accuracy and pertinence of guidance and warning, avoiding ambiguous guidance or false warnings, and providing core support for the precision and safe operation of engineering machinery.

[0103] In step S105, the operation guidance instruction is generated based on spatial relationship parameters and the required construction accuracy. It guides the operator in adjusting the working posture and position of the construction machinery, providing clear adjustment direction and amount, and can be directly used for operation guidance. The safety warning information is generated based on spatial relationship parameters and the safety risk level, reminding the operator to avoid danger. It includes the risk level, cause of danger, and countermeasures, enabling early warning and timely reminders of safety risks.

[0104] The core of this step is: based on the spatial relationship parameters determined in S104, combined with preset construction accuracy thresholds and safety risk thresholds, the parameters are graded and analyzed to generate operation guidance instructions adapted to construction accuracy requirements and safety warning information adapted to safety risk levels, thereby achieving precise guidance and safety protection for construction machinery operations. Optionally, step S105 includes: judging the current operation status based on the deviation of the normal distance between the key points of the end-effector and the design surface, and the deviation between the current position of the end-effector and the target construction outline or preset operation path, and generating operation guidance instructions including adjustment direction and adjustment amount; judging the safety risk level based on the minimum distance between the envelope of the end-effector and the operation surface, the safe distance between the construction machinery body or the end-effector and the dangerous area, and the contact, crossing, and overlapping status; and outputting corresponding level safety warning information and safety control instructions for deceleration, stopping, and moving away from the dangerous area according to the operation status and safety risk level.

[0105] In practice, firstly, construction accuracy thresholds and safety risk thresholds are preset. The construction accuracy thresholds correspond to construction design requirements (such as elevation deviation thresholds and slope deviation thresholds), while the safety risk thresholds correspond to safety constraint requirements (such as minimum safe distance thresholds and intrusion risk level thresholds).

[0106] For generating operation guidance instructions: read the spatial relationship parameters of the operation end mechanism, operation surface and construction design information in S104, and compare them with the preset construction accuracy threshold; if the parameters do not exceed the accuracy threshold, generate an instruction to maintain the current operation posture; if the parameters exceed the accuracy threshold, generate a clear adjustment instruction based on the deviation direction and deviation magnitude. The instruction includes the adjustment direction (under excavation, uplift, leftward movement, rightward movement) and adjustment amount. At the same time, combined with the operation scenario, supplementary operation assistance prompts (such as adjusting to the design elevation) are provided to ensure that the operator can directly complete the operation adjustment according to the instructions.

[0107] For generating safety warning information: the spatial relationship parameters of the end-of-operation mechanism, working surface, and safety constraint information in S104 are read and compared with the preset safety risk threshold to classify the risk level (e.g., no risk, low risk, medium risk, high risk). When there is no risk, no warning information is generated; when there is low risk, a prompt warning is generated (e.g., approaching a dangerous area, please operate with caution); when there is medium risk, a warning warning is generated (e.g., about to enter a dangerous area, please slow down and adjust); when there is high risk, an emergency warning is generated (e.g., have already entered a dangerous area, please stop work immediately and evacuate), while also indicating the cause of the danger (e.g., less than 1m from underground pipelines) and the safety distance requirement, providing clear response guidance for the operator.

[0108] The beneficial effects of step S105 are as follows: it transforms the quantified spatial relationship parameters into directly executable operation guidance instructions and hierarchical safety early warning information, realizing precise guidance and proactive early warning for construction machinery operations; it effectively solves the problems of ambiguous operation guidance and inability to predict safety risks in advance in existing technologies, which can not only improve construction accuracy and reduce construction errors such as over-excavation, under-excavation, and slope deviation, but also proactively avoid safety hazards such as collisions and intrusion into dangerous areas, reduce operation risks, and ensure the safety of operators, equipment and construction sites, while improving operation efficiency and achieving the dual goals of precise construction and safe construction.

[0109] In this embodiment of the invention, the entire process of construction machinery operation is achieved through closed-loop management, from multi-source data acquisition and scene modeling to precise guidance and safety early warning. By integrating multi-source sensing data and multi-layer semantic modeling, the problems of low accuracy of single data sources, fragmented scene information, and inconsistent benchmarks of multi-source data are solved, realizing the digitization of operation scenarios and the quantification of status. Through precise spatial relationship calculation and hierarchical guidance and early warning, construction accuracy is effectively improved, construction errors are reduced, safety hazards are proactively avoided, operational risks are reduced, and the safety of operators, equipment, and construction sites is ensured. At the same time, operational efficiency is improved, achieving the goal of precise, safe, and efficient operation of construction machinery.

[0110] Example 2 Figure 2 A flowchart of a method for guiding the operation of construction machinery is provided as an embodiment of the present invention. See also... Figure 2 The method includes the following steps: S201. Acquire multi-source sensing data, prior geometric parameters of equipment, and pre-stored construction design information and safety constraint information during the operation of construction machinery; multi-source sensing data includes binocular vision data, dToF depth data, inertial measurement data, and global positioning data.

[0111] S202. Perform deep learning adaptive fusion processing on binocular vision data and dToF depth data to construct a continuous three-dimensional point cloud model of the near-field environment of the operation.

[0112] S203. While constructing a continuous 3D point cloud model, the system integrates binocular vision data, dToF depth data, inertial measurement data, and global positioning data for joint optimization processing to determine the continuous pose of the device.

[0113] S204. Based on the prior geometric parameters of the equipment and the features of the end-effector extracted from the continuous three-dimensional point cloud model or binocular vision data, calculate the state information of the end-effector of the construction machinery; the state information includes joint angles, key point coordinates and envelope model.

[0114] S205. Construct a geometric model of the work surface based on a continuous three-dimensional point cloud model. Relying on continuous pose, convert the state information of the end mechanism and the geometric model of the work surface to the world coordinate system. Then, uniformly register the construction design information and safety constraint information to the world coordinate system to construct a multi-layer semantic model of the work scene.

[0115] S206. Based on the semantic model of multi-layer operation scenarios, determine the spatial relationship parameters between the end-of-operation mechanism, the working surface, and the construction design information and safety constraint information, respectively.

[0116] S207. Based on spatial relationship parameters, generate operation guidance instructions and safety warning information for construction machinery.

[0117] The specific implementation of steps S201-S207 can be found in the description of the above embodiments, and will not be repeated here.

[0118] S208. Convert the construction design information, the status information of the end-of-operation mechanism, the geometric model of the working surface, the safety constraint information, the operation guidance instructions and the safety warning information from the unified world coordinate system to the screen coordinate system or the augmented reality coordinate system; and overlay and display the design surface, target path, end-of-operation posture, deviation value, dangerous boundary and safe distance through the cab terminal, mobile terminal or augmented reality device to build an augmented reality visual operation guidance interface.

[0119] Specifically, firstly, various types of data that have been uniformly mapped to the world coordinate system in the multi-layer operation scenario semantic model constructed by S205 are retrieved, including construction design information of the design surface layer (3D model of the design surface, target construction outline, design elevation, etc.), status information of the operation end mechanism of the equipment layer (joint angle, key point coordinates, envelope model), geometric model of the operation surface of the operation surface layer, and safety constraint information of the dangerous area layer (dangerous area boundary, underground pipeline range, etc.), to ensure that all data to be displayed has a consistent spatial reference and compatible format.

[0120] Secondly, various types of data are visualized and rendered: different rendering methods are used to distinguish different types of data. The geometric model of the work surface is rendered with textures that fit the actual site. The construction design information is rendered with highlighted lines or semi-transparent overlays. The envelope of the end mechanism is rendered with eye-catching colors (such as blue) to highlight it. The dangerous areas in the safety constraint information are rendered with warning colors (such as red) and the hazard type is marked. The key points of the end mechanism are highlighted and rendered to ensure that all types of information are clearly distinguishable and do not obscure each other.

[0121] Simultaneously, the operation guidance instructions and safety warning information generated by S207 are retrieved, and the operation guidance instructions (including adjustment direction and adjustment amount) are overlaid on the corresponding operation area (such as near the end mechanism of the operation) in the form of clear text and icons; the safety warning information is displayed in a differentiated form according to the risk level, with low-risk warnings displayed with prompt text, and medium- and high-risk warnings displayed in the form of flashing warning boxes, eye-catching text and icons, and the cause of danger and countermeasures are marked at the same time to ensure that operators can quickly capture key information.

[0122] Finally, an augmented reality visualization operation guidance interface is built, which accurately overlays and aligns the rendered construction design information, the status information of the end-of-line mechanism, the geometric model of the work surface, and the safety constraint information with the real-time image of the work site, so as to realize the integrated display of virtual design data, equipment status data and actual on-site scene; the interface also integrates guidance instructions and early warning information display areas, supports real-time information updates, and is adapted to the display terminal of the construction machinery cab, so as to ensure that the operator can intuitively obtain all key information without switching interfaces during the operation, thus realizing visualization operation guidance.

[0123] In this embodiment of the invention, by overlaying and visualizing multi-source data to build an augmented reality guidance interface, the problem of scattered presentation and difficulty in intuitive association of information such as construction design, equipment status, and safety constraints in the prior art is solved. This achieves precise integration of virtual design data and actual on-site scenarios. The synchronous and intuitive presentation of operation guidance instructions and safety warning information reduces the difficulty for operators to interpret complex data, making it easier for operators to quickly understand operation deviations and safety risks, adjust their operation posture in a timely manner according to guidance instructions, and respond quickly to safety warnings, further improving operation efficiency and construction accuracy. At the same time, the clear visualization can reduce operator misjudgment, effectively avoid safety hazards, ensure operation safety, lower the operation threshold, and improve the convenience and intelligence level of construction machinery operation.

[0124] Example 3 Figure 3 This is a schematic diagram of a construction machinery operation guidance device provided in an embodiment of the present invention. This device is equipped to execute any of the construction machinery operation guidance methods of the present invention. For example... Figure 3As shown, the construction machinery operation guidance device includes: The data acquisition module 301 is used to acquire the prior geometric parameters of the equipment, pre-stored construction design information and safety constraint information, a three-dimensional point cloud model of the near-field environment of the operation, and the continuous pose of the equipment when the construction machinery starts to operate; wherein, the three-dimensional point cloud model of the near-field environment of the operation and the continuous pose of the equipment are determined and output by an integrated fusion sensing device installed on the construction machinery based on the collected multi-source sensing data; the multi-source sensing data includes at least binocular vision data, dToF depth data, and inertial measurement data; The end-effector state determination module 302 is used to calculate the state information of the end-effector of the construction machinery based on the prior geometric parameters of the equipment and the features of the end-effector extracted from the continuous three-dimensional point cloud model or binocular vision data; the state information includes joint angles, key point coordinates and envelope model; The model building module 303 is used to build a geometric model of the working surface based on a continuous three-dimensional point cloud model. It converts the state information of the end mechanism and the geometric model of the working surface to the world coordinate system based on the continuous pose of the equipment, and uniformly registers the construction design information and safety constraint information to the world coordinate system, thereby building a multi-layer semantic model of the working scene. The spatial relationship determination module 304 is used to determine the spatial relationship parameters between the end-of-operation mechanism and the working surface and the construction design information and safety constraint information, respectively, based on the multi-layer operation scenario semantic model. The operation guidance module 305 is used to generate operation guidance instructions and safety warning information for construction machinery based on spatial relationship parameters.

[0125] In some embodiments, it also includes: The fusion processing module is used to obtain a continuous three-dimensional point cloud model of the near-field environment of the operation and the corresponding point cloud fusion confidence based on the fusion processing of the binocular vision data and the dToF depth data. The continuous pose determination module is used to simultaneously fuse the binocular vision data, the dToF depth data, and the inertial measurement data for joint optimization processing while constructing the continuous 3D point cloud model, so as to determine the continuous pose of the device.

[0126] In some embodiments, the model building module 303 is specifically used for: From a continuous 3D point cloud model, the target area point cloud corresponding to the construction scene is segmented and extracted; The point cloud of the target area is fitted or meshed to generate a geometric model of the working surface that matches the construction scene. Based on the continuous pose of the equipment, the status information of the end-effector and the geometric model of the working surface are converted to the world coordinate system, and the construction design information and safety constraint information are uniformly registered to the world coordinate system. The unified mapped continuous pose of the equipment and the status information of the end-effector mechanism are incorporated into the equipment layer, the geometric model of the working surface is incorporated into the working surface layer, the construction design information is incorporated into the design surface layer, and the safety constraint information is incorporated into the hazardous area layer to construct a multi-layer semantic model of the working scenario. The safety constraint information includes at least one of the following: underground pipelines, high-voltage lines, dangerous slope zones, restricted areas for personnel, restricted areas for equipment, and dynamic safety boundaries.

[0127] In some embodiments, the spatial relationship determination module 304 is specifically used for: Retrieve data corresponding to the equipment layer, work surface layer, design surface layer, and hazardous area layer in the multi-layer operation scenario semantic model; Based on the data corresponding to the equipment layer and the design surface layer, determine the normal distance deviation between the key points of the end-of-operation mechanism and the design surface in the construction design information, as well as the spatial position deviation and attitude deviation between the current position of the end-of-operation mechanism and the target construction outline and preset operation path in the construction design information. Based on the data corresponding to the equipment layer and the working surface layer, determine the minimum distance and contact state between the working end mechanism envelope and the working surface, which is constructed from the key points, edge points and simplified geometric model of the working end mechanism; the type of the working end mechanism envelope includes at least one of convex hull, bounding box, capsule body or mesh envelope. Based on the data corresponding to the equipment layer and the hazardous area layer, the safe distance and spatial intrusion status between the engineering machinery body, the end-effector, and the hazardous area defined by the safety constraint information are determined.

[0128] In some embodiments, the fusion processing module is specifically used for: The binocular vision data is corrected and disparity is estimated to generate a binocular depth map. The corresponding binocular confidence is determined based on left-right consistency, matching cost, texture intensity, edge stability and temporal stability. The dToF depth data is denoised and multipath suppressed to generate a dToF depth map. The corresponding dToF confidence is determined based on echo intensity, signal-to-noise ratio, multipath risk, flying point probability, neighborhood depth consistency and historical stability. Based on a deep learning model, and combining scene and data feature parameters, as well as binocular confidence and dToF confidence, the pixel-wise fusion weights corresponding to the binocular depth map and the dToF depth map are determined respectively; wherein, the scene and data features include at least one feature among texture, reflectivity, distance, viewpoint change, dust / occlusion, and historical stability. Based on the pixel-wise fusion weights of the stereo depth map and the dToF depth map, an adaptive weighted fusion process is performed on the stereo depth map and the dToF depth map to obtain fused depth data and determine the corresponding point cloud fusion confidence. The fused depth data is combined with the point cloud fusion confidence score to perform voxel filtering and outlier removal to generate a continuous 3D point cloud model of the near-field environment of the operation.

[0129] In some embodiments, the multi-source sensing data further includes global positioning data, which includes at least one of the following: high-precision GNSS positioning data based on RTK, total station measurement data, UWB positioning data, or positioning data obtained by calculating the coordinates of construction control points; The continuous pose determination module is also used for: While constructing the continuous 3D point cloud model, the binocular vision data, dToF depth data, inertial measurement data, and global positioning data are integrated for joint optimization processing to determine the continuous pose of the device.

[0130] In some embodiments, the continuous pose determination module is specifically used for: Extracting image features and depth features from binocular vision data and dToF depth data; Based on image and depth features, the relative pose changes between adjacent frames are estimated through inter-frame registration, iterative nearest point matching, or feature matching. Pre-integration processing is performed on the inertial measurement data to generate incremental motion constraint data between adjacent frames; When the global positioning data is valid, the global coordinate information it provides is used as a constraint for pose estimation; When the global positioning data degrades or becomes invalid, its constraint strength is reduced or the constraint term is no longer used; By employing sliding window optimization, extended Kalman filtering, or factor graph optimization methods, the relative pose change, incremental motion constraint data, and pose estimation constraint terms are jointly estimated to determine the continuous pose of the device.

[0131] In some embodiments, the end-effector state determination module 302 is specifically used for: Based on the prior geometric parameters of the equipment, a linkage kinematic model of the engineering machinery is constructed; Extract key geometric features of the end-effector from continuous 3D point cloud models or binocular vision data. Key geometric features include key points of the mechanism, edges of links, or end contours. If the construction machinery is equipped with joint sensors, CAN bus data, or hydraulic cylinder stroke data, then such data is used as prior information or constraints; if the above data is not available, then the joint angle of the end effector is estimated based on key geometric features through key point detection, edge extraction, or geometric matching. Based on the linkage kinematics model, joint angles, and continuous pose of the equipment, the poses of the key points, end edges, and end envelope of the end-effector in the world coordinate system are solved, which serve as the state information of the end-effector.

[0132] In some embodiments, the job guidance module 305 is specifically used for: Based on the deviation of the normal distance between the key points of the end-of-work mechanism and the design surface, and the deviation between the current position of the end-of-work mechanism and the target construction outline or preset work path, the current work status is determined, and work guidance instructions including adjustment direction and adjustment amount are generated. The safety risk level is determined based on the minimum distance between the envelope of the end-effector and the working surface, the safe distance between the engineering machinery body or the end-effector and the hazardous area, and the contact, crossing, and overlapping status. Based on the operational status and safety risk level, output corresponding safety warning information and safety control commands such as deceleration, stopping, and moving away from dangerous areas.

[0133] In some embodiments, a visualization module is also included, for: The construction design information, the status information of the end-of-operation mechanism, the geometric model of the working surface, the safety constraint information, the operation guidance instructions and the safety early warning information are converted from the unified world coordinate system to the screen coordinate system or the augmented reality coordinate system; By overlaying and displaying design surfaces, target paths, end-of-operation postures, deviation values, hazardous boundaries, and safe distances through cab terminals, mobile terminals, or augmented reality devices, an augmented reality visualization operation guidance interface is constructed.

[0134] In some embodiments, the end-of-work mechanism includes at least one of an excavator boom, stick, bucket, bucket teeth, bucket tip, and bucket edge.

[0135] The engineering machinery operation guidance device provided in the embodiments of the present invention can execute the engineering machinery operation guidance method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0136] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0137] Example 4 Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0138] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory 12 or a random access memory 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 12 or loaded from storage unit 18 into the random access memory 13. The random access memory 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, read-only memory 12, and random access memory 13 are interconnected via a bus 14. An input / output interface 15 is also connected to the bus 14.

[0139] Multiple components in electronic device 10 are connected to input / output interface 15, including: input unit 16; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disks, optical disks, etc.; and communication unit 19, such as network interface cards, modems, wireless transceivers, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as performing engineering machinery operation guidance methods.

[0141] In some embodiments, the construction machinery operation guidance method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via read-only memory 12 and / or communication unit 19. When the computer program is loaded into random access memory 13 and executed by processor 11, one or more steps of the construction machinery operation guidance method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the construction machinery operation guidance method by any other suitable means (e.g., by means of firmware).

[0142] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), complex programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable engineering machinery operation guidance device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device or liquid crystal display for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet. The computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having client-server relationships with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0147] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for guiding the operation of engineering machinery, characterized in that, include: When the construction machinery begins operation, it acquires the equipment's prior geometric parameters, pre-stored construction design information and safety constraint information, a 3D point cloud model of the near-field environment, and the continuous pose of the equipment. The 3D point cloud model of the near-field environment and the continuous pose of the equipment are determined and output by an integrated fusion sensing device installed on the construction machinery based on collected multi-source sensing data. The multi-source sensing data includes at least binocular vision data, dToF depth data, and inertial measurement data. Based on the prior geometric parameters of the equipment and the features of the end-effector extracted from the continuous 3D point cloud model or the binocular vision data, the state information of the end-effector of the construction machinery is calculated; the state information includes joint angles, key point coordinates and envelope model. Based on the continuous 3D point cloud model, a geometric model of the working surface is constructed. The state information of the end-effector and the geometric model of the working surface are transformed into the world coordinate system based on the continuous pose of the equipment. The construction design information and safety constraint information are uniformly registered to the world coordinate system to construct a multi-layer semantic model of the working scene. Based on the multi-layered operation scenario semantic model, spatial relationship parameters between the operation end mechanism, the operation surface, and the construction design information and safety constraint information are determined respectively. Based on the spatial relationship parameters, operation guidance instructions and safety warning information for the engineering machinery are generated.

2. The method according to claim 1, characterized in that, The process by which the integrated fusion sensing device determines and outputs a 3D point cloud model of the near-field environment and the continuous pose of the device based on the collected multi-source sensing data includes: Based on the fusion processing of the binocular vision data and the dToF depth data, a continuous three-dimensional point cloud model of the near-field environment of the operation and the corresponding point cloud fusion confidence are obtained. While constructing the continuous 3D point cloud model, the binocular vision data, the dToF depth data, and the inertial measurement data are simultaneously fused and jointly optimized to determine the continuous pose of the device.

3. The method according to claim 1, characterized in that, The process involves constructing a work surface geometric model based on the continuous 3D point cloud model, transforming the end-effector state information and the work surface geometric model to the world coordinate system using the continuous pose of the equipment, and uniformly registering the construction design information and the safety constraint information to this world coordinate system. This process constructs a multi-layered work scene semantic model, including: From the continuous three-dimensional point cloud model, the target area point cloud corresponding to the construction scene is segmented and extracted; The point cloud of the target area is fitted or meshed to generate a geometric model of the working surface that matches the construction scene. Based on the continuous pose of the equipment, the status information of the end-effector and the geometric model of the working surface are transformed into the world coordinate system, and the construction design information and the safety constraint information are uniformly registered to the world coordinate system. The unified mapped continuous pose of the equipment and the state information of the end-effector are assigned to the equipment layer, the geometric model of the working surface is assigned to the working surface layer, the construction design information is assigned to the design surface layer, and the safety constraint information is assigned to the hazardous area layer to construct a multi-layer semantic model of the working scenario. The safety constraint information includes at least one of the following: underground pipelines, high-voltage lines, dangerous slope zones, restricted areas for personnel, restricted areas for equipment, and dynamic safety boundaries.

4. The method according to claim 3, characterized in that, The determination of spatial relationship parameters between the end-of-operation mechanism, the work surface, and the construction design information and safety constraint information, based on the multi-layered operational scenario semantic model, includes: Retrieve data corresponding to the equipment layer, work surface layer, design surface layer, and hazardous area layer in the multi-layered operation scenario semantic model; Based on the data corresponding to the equipment layer and the design surface layer, determine the normal distance deviation between the key points of the end-of-operation mechanism and the design surface in the construction design information, as well as the spatial position deviation and posture deviation between the current position of the end-of-operation mechanism and the target construction outline and preset operation path in the construction design information; Based on the data corresponding to the equipment layer and the working surface layer, the minimum distance and contact state between the working end mechanism envelope, constructed from the key points, edge points, and simplified geometric model of the working end mechanism, and the working surface are determined; the type of the working end mechanism envelope includes at least one of convex hull, bounding box, capsule body, or mesh envelope. Based on the data corresponding to the equipment layer and the hazardous area layer, the safe distance and spatial intrusion status between the engineering machinery body, the end-of-line mechanism, and the hazardous area defined by the safety constraint information are determined.

5. The method according to claim 2, characterized in that, The process of obtaining a continuous 3D point cloud model of the near-field environment of the operation and the corresponding point cloud fusion confidence score based on the fusion processing of the binocular vision data and the dToF depth data includes: The binocular vision data is corrected and disparity is estimated to generate a binocular depth map. The corresponding binocular confidence is determined based on left-right consistency, matching cost, texture intensity, edge stability and temporal stability. The dToF depth data is denoised and multipath suppressed to generate a dToF depth map, and the corresponding dToF confidence is determined based on echo intensity, signal-to-noise ratio, multipath risk, flying point probability, neighborhood depth consistency and historical stability. Based on a deep learning model, and combining scene and data feature parameters with the binocular confidence and the dToF confidence, the pixel-wise fusion weights corresponding to the binocular depth map and the dToF depth map are determined respectively; wherein, the scene and data features include at least one feature among texture, reflectivity, distance, viewpoint change, dust / occlusion, and historical stability. Based on the pixel-wise fusion weights of the stereo depth map and the dToF depth map, an adaptive weighted fusion process is performed on the stereo depth map and the dToF depth map to obtain fused depth data and determine the corresponding point cloud fusion confidence. The fused depth data is combined with the point cloud fusion confidence score to perform voxel filtering and outlier removal processing to generate a continuous three-dimensional point cloud model of the near-field environment of the operation.

6. The method according to claim 1, characterized in that, The multi-source sensing data also includes global positioning data, which includes at least one of the following: high-precision GNSS positioning data based on RTK, total station measurement data, UWB positioning data, or positioning data obtained by calculating the coordinates of construction control points; The method further includes: While constructing the continuous 3D point cloud model, the binocular vision data, dToF depth data, inertial measurement data, and global positioning data are integrated for joint optimization processing to determine the continuous pose of the device.

7. The method according to claim 6, characterized in that, While constructing the continuous 3D point cloud model, the binocular vision data, dToF depth data, inertial measurement data, and global positioning data are fused together for joint optimization processing to determine the continuous pose of the device, including: Image features and depth features are extracted from the binocular vision data and the dToF depth data; Based on the image features and depth features, the relative pose changes between adjacent frames are estimated through inter-frame registration, iterative nearest point matching, or feature matching. The inertial measurement data is pre-integrated to generate incremental motion constraint data between adjacent frames; When the global positioning data is valid, the global coordinate information it provides is used as a constraint for pose estimation; When the global positioning data degrades or becomes invalid, its constraint strength is reduced or the constraint term is no longer used; The continuous pose of the device is determined by jointly estimating the relative pose change, incremental motion constraint data, and pose estimation constraint terms using sliding window optimization, extended Kalman filtering, or factor graph optimization methods.

8. The method according to claim 1, characterized in that, The process of calculating the state information of the end-effector of the construction machinery based on the prior geometric parameters of the equipment and the features of the end-effector extracted from the continuous 3D point cloud model or the binocular vision data includes: Based on the prior geometric parameters of the equipment, a linkage kinematic model of the engineering machinery is constructed; Extract key geometric features of the end-effector mechanism from the continuous 3D point cloud model or the binocular vision data. The key geometric features include key points of the mechanism, edges of the members, or end contours. If the construction machinery is equipped with joint sensors, CAN bus data, or hydraulic cylinder stroke data, then such data is used as prior information or constraints; if no such data is available, then the joint angle of the end-effector is estimated based on the key geometric features by key point detection, edge extraction, or geometric matching. Based on the linkage kinematic model, the joint angles, and the continuous pose of the device, the poses of the key points, end edges, and end envelope of the end-effector in the world coordinate system are solved, which serve as the state information of the end-effector.

9. The method according to claim 4, characterized in that, The step of generating operation guidance instructions and safety warning information for the construction machinery based on the spatial relationship parameters includes: Based on the deviation of the normal distance between the key points of the end-of-work mechanism and the design surface, and the deviation between the current position of the end-of-work mechanism and the target construction outline or preset work path, the current work status is determined, and work guidance instructions containing adjustment direction and adjustment amount are generated. The safety risk level is determined based on the minimum distance between the envelope of the end-effector and the working surface, the safe distance between the engineering machinery body or the end-effector and the hazardous area, and the contact, crossing, and overlapping states. Based on the operational status and safety risk level, output corresponding level of safety warning information and safety control commands such as deceleration, stopping, and moving away from the dangerous area.

10. The method according to claim 9, characterized in that, Also includes: The construction design information, the status information of the end-of-operation mechanism, the geometric model of the working surface, the safety constraint information, the operation guidance instructions and the safety early warning information are converted from the unified world coordinate system to the screen coordinate system or the augmented reality coordinate system; By overlaying and displaying design surfaces, target paths, end-of-operation postures, deviation values, hazardous boundaries, and safe distances through cab terminals, mobile terminals, or augmented reality devices, an augmented reality visualization operation guidance interface is constructed.

11. The method according to claim 1, characterized in that, The end-of-operation mechanism includes at least one of the following: excavator boom, stick, bucket, bucket teeth, bucket tip, and bucket edge.

12. A working guidance device for engineering machinery, characterized in that, include: The data acquisition module is used to acquire the prior geometric parameters of the equipment, pre-stored construction design information and safety constraint information, a 3D point cloud model of the near-field environment, and the continuous pose of the equipment when the construction machinery starts operating. The 3D point cloud model of the near-field environment and the continuous pose of the equipment are determined and output by an integrated fusion sensing device installed on the construction machinery based on collected multi-source sensing data. The multi-source sensing data includes at least binocular vision data, dToF depth data, and inertial measurement data. The end-effector state determination module is used to calculate the state information of the end-effector of the construction machinery based on the prior geometric parameters of the equipment and the end-effector features extracted from the continuous three-dimensional point cloud model or the binocular vision data; the state information includes joint angles, key point coordinates and envelope model; The model building module is used to build a geometric model of the working surface based on the continuous three-dimensional point cloud model, and to convert the state information of the end mechanism and the geometric model of the working surface to the world coordinate system based on the continuous pose of the equipment. The construction design information and the safety constraint information are uniformly registered to the world coordinate system to build a multi-layer semantic model of the working scene. The spatial relationship determination module is used to determine the spatial relationship parameters between the end-of-operation mechanism, the working surface, and the construction design information and the safety constraint information, respectively, based on the multi-layer operation scenario semantic model. The operation guidance module is used to generate operation guidance instructions and safety warning information for the construction machinery based on the spatial relationship parameters.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to perform the method of any one of claims 1-11.

Images