Three-dimensional reconstruction method and device for operation equipment and operation equipment

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method and device for operation equipment and the operation equipment, and relates to the technical field of engineering machinery. A measuring device is arranged on the operation equipment. The method comprises the following steps: acquiring real-time pose data of the operation equipment and real-time point cloud acquired by the measuring device; determining the real-time position of the measuring device under the operation equipment coordinate system according to the real-time pose data; according to the pose data and the real-time position of the key component in the real-time pose data, the region of interest of the key component is determined in the real-time point cloud, so that the calculation amount of three-dimensional reconstruction based on the method can be reduced; and filtering artifact points in the real-time point cloud based on a dynamic point cloud filtering algorithm to obtain a processed point cloud without artifacts for three-dimensional reconstruction of operation equipment. According to the method, artifacts caused by motion of the operation equipment can be eliminated, and artifact points generated by motion of an environment object are eliminated by filtering artifact points in combination with a dynamic point cloud filtering algorithm, so that three-dimensional reconstruction can be carried out through the processed point cloud.

Description

Technical Field

[0001] This application relates to the field of engineering machinery technology, specifically to a three-dimensional reconstruction method, device, and operating equipment for working equipment. Background Technology

[0002] With the increasing complexity of large-scale engineering construction and the ever-increasing safety requirements, the intelligent and unmanned operation of crane equipment has become an important development direction. Its core relies on accurate and real-time perception of the working environment (including static structures and dynamic targets). LiDAR (Light Detection and Ranging) and visual sensors are key means of acquiring 3D point cloud data of the environment and are widely used in obstacle detection and avoidance systems. However, for large-scale rotating and luffing equipment like cranes, dynamic obstacles (such as moving vehicles, personnel, and temporary equipment) easily produce "motion artifacts" in continuously scanned point cloud data, severely interfering with the accurate reconstruction of static scenes (such as building structures and fixed equipment). Existing point cloud-based static environment reconstruction methods struggle to effectively distinguish and filter out these dynamic targets and their motion artifacts, leading to model distortion and decreased accuracy. Therefore, a technical solution is needed to eliminate "motion artifacts" during 3D model reconstruction of engineering machinery operating scenarios to improve model accuracy. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, and working equipment for three-dimensional reconstruction of working equipment.

[0004] To achieve the above objectives, the first aspect of this application provides a three-dimensional reconstruction method for working equipment, wherein the working equipment is equipped with a measuring device; the three-dimensional reconstruction method includes: Acquire real-time pose data of the operating equipment and real-time point cloud data collected by the measuring device; The real-time position of the measuring device in the coordinate system of the working equipment is determined based on the real-time pose data. Based on the pose data and real-time position of key components in the real-time pose data, the region of interest of the key components is determined in the real-time point cloud, and the artifact points in the real-time point cloud are filtered out based on the dynamic point cloud filtering algorithm to obtain the processed point cloud with artifacts removed for use in the 3D reconstruction of the operating equipment.

[0005] In this embodiment of the application, the process of determining the region of interest (ROI) of the key component in the real-time point cloud based on the pose data and real-time position of the key component in the real-time pose data, and filtering out artifact points in the real-time point cloud based on a dynamic point cloud filtering algorithm to obtain a processed point cloud with artifacts removed for use in the 3D reconstruction of the working equipment includes: determining the ROI of the key component in the real-time point cloud based on the pose data and real-time position of the key component in the real-time pose data, obtaining point cloud clusters within the ROI; and filtering out artifact points in the point cloud clusters within the ROI based on a dynamic point cloud filtering algorithm.

[0006] In this embodiment of the application, artifact points include discrete points. Filtering artifact points in real-time point clouds based on dynamic point cloud filtering algorithm includes: filtering out sampling points in real-time point clouds when the local density of any sampling point in the real-time point cloud is less than a preset local density threshold.

[0007] In this embodiment, artifact points include motion anomaly points, and the dynamic point cloud filtering algorithm includes a temporal consistency check algorithm; filtering artifact points in real-time point clouds based on the dynamic point cloud filtering algorithm includes: filtering motion anomaly points in real-time point clouds according to the temporal consistency check algorithm.

[0008] In this embodiment, filtering out motion anomalies in real-time point clouds using a temporal consistency check algorithm includes: acquiring historical point clouds, where the acquisition time of the historical point clouds is earlier than that of the real-time point clouds; determining real-time point cloud clusters obtained by clustering the real-time point clouds and historical point cloud clusters obtained by clustering the historical point clouds based on a clustering algorithm; determining historical point cloud clusters that match the real-time point cloud clusters based on a combinatorial optimization algorithm, as the matching result of the real-time point cloud clusters; determining the motion amount of the real-time point cloud clusters from the generation time of the historical point clouds to the generation time of the real-time point clouds based on the matching result; and identifying the real-time point cloud clusters as motion anomalies and filtering them out from the real-time point clouds if the motion amount exceeds a preset motion amount range.

[0009] In this embodiment of the application, the combinatorial optimization algorithm is the Hungarian algorithm; the historical point cloud clusters that match the real-time point cloud clusters are determined based on the combinatorial optimization algorithm, and the matching result of the real-time point cloud clusters includes: determining the historical point cloud clusters that match the real-time point cloud clusters based on the association cost formula of the Hungarian algorithm, and the matching result of the real-time point cloud clusters, wherein the association cost formula includes the distance cost, shape matching cost and volume matching cost of the two point cloud clusters.

[0010] In this embodiment of the application, the association cost formula (1) is:

[0011] (1) in, Represents a real-time point cloud cluster. Represents historical point cloud clusters, This represents the association cost between two cloud clusters. This represents the distance cost between two cloud clusters. This represents the cost of shape matching between two cloud clusters. This represents the cost of matching the volume of two cloud clusters. , and These represent the weighting coefficients of each cost item.

[0012] In this embodiment of the application, obtaining real-time pose data of the working equipment includes: obtaining pose data of key components, wherein the pose data of key components is determined based on the historical motion commands of the key components and the motion model of the working equipment.

[0013] In this embodiment of the application, before acquiring the pose data of the key component, the 3D reconstruction method further includes: acquiring the motion command of the key component; determining the motion trajectory of the key component within a preset time period based on the motion command and the motion model of the operating equipment, wherein the motion trajectory includes the predicted pose data of the key component at multiple different timestamps; acquiring the pose data of the key component includes: selecting the predicted pose data corresponding to the first timestamp of the real-time pose data or the second timestamp that is the same as the first timestamp of the real-time point cloud as the pose data of the key component in the motion trajectory.

[0014] In this embodiment of the application, the working equipment is further equipped with an image acquisition device, and the three-dimensional reconstruction method further includes: acquiring real-time images captured by the image acquisition device; determining the relative positions of the measuring device and the image acquisition device in the coordinate system of the working equipment based on real-time pose data; determining the position of the object mask in the real-time image in the real-time point cloud based on the relative position, and determining the object point cloud cluster in the real-time point cloud based on the object mask; determining the region of interest of the key component in the real-time point cloud based on the pose data and real-time position of the key component in the real-time pose data, and filtering out artifact points in the real-time point cloud based on the dynamic point cloud filtering algorithm, including: determining the region of interest of the key component in the object point cloud cluster based on the pose data and real-time position of the key component in the real-time pose data, and filtering out artifact points in the object point cloud cluster based on the dynamic point cloud filtering algorithm.

[0015] In this embodiment, the working device includes a motion node, and a pose sensor is provided on the motion node. The real-time pose data includes the real-time node pose data of the motion node. Obtaining the real-time pose data of the working device includes: obtaining the node pose of the motion node through the pose sensor. Determining the real-time position of the measuring device in the coordinate system of the working device based on the real-time pose data includes: determining the real-time position of the measuring device in the coordinate system of the working device based on the motion model of the working device using the real-time node pose data. The motion model includes the positional relationship between the measuring device and the motion node, and the coordinate system of the working device is set based on the motion node.

[0016] A second aspect of this application provides a three-dimensional reconstruction apparatus for a work equipment, comprising: a processor configured to retrieve instructions from memory and, when executing the instructions, to implement the three-dimensional reconstruction method for a work equipment provided in the first aspect of this application.

[0017] A third aspect of this application provides a work device, comprising: a work device body; a three-dimensional reconstruction device for the work device according to a second aspect of this application; a measuring device disposed on the work device body; and a pose sensor disposed on the work device body, the pose sensor being used to acquire real-time pose data of the work device.

[0018] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned three-dimensional reconstruction method for a working device.

[0019] By determining the region of interest using the above technical solution, the 3D reconstruction of the working equipment can be concentrated on the region of interest, thereby reducing the amount of computation required for 3D reconstruction. Since the determination of the region of interest is combined with the real-time position of the measuring device in the coordinate system of the working equipment, the position of each point in the real-time point cloud can be represented based on the equipment coordinate system. This eliminates the point cloud changes caused by the movement of the working equipment itself, and determines the accurate positional relationship between each point in the real-time point cloud and the key components. This allows the accurate determination of the region of interest in the real-time point cloud based on the pose data of the key components to eliminate artifacts caused by the movement of the working equipment itself. By combining this with a dynamic point cloud filtering algorithm to filter out artifact points, artifact points caused by the movement of environmental objects can be further eliminated. Thus, 3D reconstruction can be performed through the processed point cloud to accurately capture and predict the trajectory of dynamic targets, enabling the working equipment to effectively avoid them.

[0020] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The schematic diagram illustrates a structural schematic of a working device according to an embodiment of this application; Figure 2 The illustration shows a flowchart of a three-dimensional reconstruction method for working equipment according to an embodiment of this application; Figure 3The illustration shows a schematic flowchart of another three-dimensional reconstruction method for working equipment according to an embodiment of this application; Figure 4 The illustration shows a schematic flowchart of another three-dimensional reconstruction method for working equipment according to an embodiment of this application; Figure 5 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0023] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0024] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0025] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant laws and regulations. Furthermore, it should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0026] To achieve 3D reconstruction, work equipment requires LiDAR and vision sensors. During the 3D reconstruction process, large rotating and luffing work equipment, such as cranes, is prone to "motion artifacts" (trailing shadows) caused by dynamic obstacles (such as moving vehicles, personnel, and temporary equipment) in the continuously scanned point cloud data. This significantly interferes with the accurate reconstruction of static scenes (such as building structures and fixed equipment). Specific analysis reveals that the root cause of these artifacts lies in the continuous movement of the LiDAR on the work equipment and the surrounding objects. This causes artifacts to appear in any single frame of point cloud captured by the LiDAR due to the aforementioned motion.

[0027] Existing point cloud-based static environment reconstruction methods struggle to effectively distinguish and filter out dynamic moving targets and their motion artifacts, leading to model distortion and decreased accuracy. Therefore, to effectively eliminate motion artifacts, this application provides a 3D reconstruction method for work equipment. This method is a technical solution that can eliminate motion artifacts and improve model accuracy during the 3D model reconstruction process of the work equipment's work scene.

[0028] Figure 1 The diagram illustrates a planar structural schematic of a work device according to an embodiment of this application. The work device may be, for example, a crane, and the region of interest (ROI) of the work device may be, for example, a planar sector-shaped region (corresponding to a three-dimensional conical region) containing the hook and its load. The three-dimensional reconstruction method for work devices provided in this application embodiment can be applied to, for example... Figure 1 The three-dimensional reconstruction process of the operating equipment shown.

[0029] Figure 2 A schematic flowchart illustrating a three-dimensional reconstruction method for working equipment according to an embodiment of this application is shown. Figure 2 As shown, in one embodiment of this application, a three-dimensional reconstruction method for working equipment is provided. This embodiment mainly uses this method to reconstruct, for example, Figure 1 The following example illustrates the 3D reconstruction of a work equipment. The work equipment is equipped with a measuring device used to collect real-time point clouds. The 3D reconstruction method for work equipment provided in this embodiment includes the following steps: S202. Acquire real-time pose data of the operating equipment and real-time point cloud data collected by the measuring device.

[0030] Understandably, the real-time pose data of the working equipment may include the real-time pose data of multiple components within the working equipment, including measuring devices. These measuring devices may be, for example, lidar, depth cameras, 3D scanners, or other measuring devices capable of generating point clouds.

[0031] S204. Determine the real-time position of the measuring device in the coordinate system of the working equipment based on the real-time pose data; S206. Based on the pose data and real-time position of key components in the real-time pose data, determine the region of interest of key components in the real-time point cloud, and filter out artifact points in the real-time point cloud based on the dynamic point cloud filtering algorithm to obtain the processed point cloud with artifacts removed for use in the 3D reconstruction of the operating equipment.

[0032] The 3D reconstruction method for work equipment provided in this application determines the real-time position of the measuring device in the work equipment coordinate system by using the real-time pose data of the work equipment. This establishes a connection between the real-time point cloud and the work equipment coordinate system. The real-time pose data includes the pose data of key components of the work equipment. Since a connection has been established between the real-time point cloud and the work equipment coordinate system, based on the pose data of the key component and the aforementioned real-time position, it is possible to determine which part of the point cloud in the real-time point cloud corresponds to the point cloud near the key component. This allows for the division of the region of interest of the key component in the real-time point cloud. Furthermore, a dynamic point cloud filtering algorithm is used to filter out artifact points in the real-time point cloud, resulting in a processed point cloud with artifacts removed for use in the 3D reconstruction of the work equipment. In the above process, the determination of the region of interest (ROI) allows the 3D reconstruction of the working equipment to be concentrated in the ROI, thereby reducing the computational load of 3D reconstruction. Since the determination of the ROI is combined with the real-time position of the measuring device in the coordinate system of the working equipment, the position of each point in the real-time point cloud can be represented based on the equipment coordinate system. This eliminates the point cloud changes caused by the movement of the working equipment itself, and determines the accurate positional relationship between each point in the real-time point cloud and the key components. This allows the accurate determination of the ROI in the real-time point cloud based on the pose data of the key components to eliminate artifacts caused by the movement of the working equipment itself. Combined with the dynamic point cloud filtering algorithm to filter out artifact points, artifact points caused by the movement of environmental objects can be further eliminated. Thus, 3D reconstruction can be performed through the processed point cloud to accurately capture and predict the trajectory of dynamic targets, enabling the working equipment to effectively avoid them.

[0033] like Figure 3 As shown, in some embodiments of this application, to further reduce the computational load of the three-dimensional reconstruction method for the working equipment, step S206 may include: S302. Based on the pose data and real-time position of key components in the real-time pose data, determine the region of interest of the key components in the real-time point cloud and obtain the point cloud clusters within the region of interest. S304. Based on the dynamic point cloud filtering algorithm, filter out artifact points in the point cloud clusters within the region of interest.

[0034] The above steps prioritize the identification of point cloud clusters within the region of interest (ROI), and then use a dynamic point cloud filtering algorithm to remove artifact points from these clusters. This concentrates computational power on the point cloud clusters within the ROI, reducing the computational load required by the method and increasing computational speed, thereby improving the real-time performance of 3D reconstruction. In practical 3D reconstruction scenarios for operational equipment, such as crane operations, general algorithms often suffer from high computational load and low efficiency. The 3D reconstruction method for operational equipment provided in this application not only eliminates artifact points in the real-time point cloud through steps S204 and S206, but also eliminates the significant contradiction between high adaptability and strict real-time requirements in complex scenes by identifying the ROI. It balances the high-precision perception requirements of complex dynamic construction site environments with the stringent processing speed requirements of operational equipment control, reducing the computational load requirements of dynamic object detection and point cloud processing algorithms, and achieving millisecond-level real-time response.

[0035] In some embodiments of this application, artifact points include discrete points. Filtering artifact points in a real-time point cloud based on a dynamic point cloud filtering algorithm includes: filtering out the sampling point in the real-time point cloud when the local density of any sampling point in the real-time point cloud is less than a preset local density threshold.

[0036] Because dynamic obstacles change position within a single frame scan time during their movement, they create "ghosting" in the real-time point cloud of that frame. This ghosting appears as a sparser and more discrete set of points (i.e., low-density regions) compared to a static background or stable object. Therefore, the artifacts caused by object motion in the real-time point cloud include discrete points, which are relatively far apart from other points in the cloud, appearing as isolated points. The above steps can filter out these discrete points, thus eliminating the artifacts.

[0037] Specifically, the dynamic point cloud filtering algorithm that removes discrete points is a geometric filtering algorithm. Removing discrete points from a real-time point cloud based on a geometric filtering algorithm may include: determining the local density of each sampling point in the real-time point cloud; and filtering out sampling points from the real-time point cloud when the local density is less than a preset local density threshold.

[0038] The above steps first determine the local density of each sampling point, thereby determining the degree of dispersion of each sampling point. If the local density of a sampling point is less than a preset local density threshold, the sampling point can be determined as a discrete point, and thus the sampling point can be filtered out from the real-time point cloud.

[0039] Specifically, the preset local density threshold can be, for example, a pre-set empirical value, or it can be determined in real time based on the local density of each sampling point in the real-time point cloud, for example, based on the mode of the local density of each sampling point.

[0040] In some embodiments of this application, artifact points may include motion anomalies, and the dynamic point cloud filtering algorithm includes a temporal consistency check algorithm; filtering artifact points in real-time point clouds based on the dynamic point cloud filtering algorithm includes: filtering motion anomalies in real-time point clouds according to the temporal consistency check algorithm.

[0041] Anomalies in motion are characterized by abnormal motion patterns of sampling points in real-time point clouds. For example, the motion speed of a certain sampling point is significantly different from that of other sampling points, or it clearly does not conform to the laws of physics. Therefore, it can be determined that anomalies in motion should not be used for 3D reconstruction. Thus, anomalies in motion can be identified as artifacts caused by the motion of objects during the point cloud acquisition process and filtered out.

[0042] In some embodiments of this application, filtering out motion anomalies in real-time point clouds based on a temporal consistency check algorithm may include: Acquire historical point clouds, which were collected earlier than real-time point clouds; Based on clustering algorithms, real-time point cloud clusters obtained from real-time point cloud clustering and historical point cloud clusters obtained from historical point cloud clustering are determined respectively; Historical point cloud clusters that match the real-time point cloud clusters are determined based on a combinatorial optimization algorithm and used as the matching result for the real-time point cloud clusters. The motion of the real-time point cloud cluster from the generation time of the historical point cloud to the generation time of the real-time point cloud is determined based on the matching results. If the amount of exercise exceeds the preset exercise range, the real-time point cloud clusters will be identified as abnormal points and filtered out from the real-time point cloud.

[0043] Clustering and combinatorial optimization algorithms can be used to match the sampling points in the real-time point cloud with those in the historical point cloud. This is equivalent to obtaining real-time point cloud clusters and historical point cloud clusters that represent the same object in the real-time and historical point clouds. The matching results are then used to determine the motion of the real-time point cloud cluster from the generation time of the historical point cloud to the generation time of the real-time point cloud. This motion represents the object's motion behavior. Based on the relationship between this motion and a preset motion interval, it is determined whether the real-time point cloud cluster is a motion anomaly. Real-time point cloud clusters that are motion anomalies are then filtered out from the real-time point cloud.

[0044] Specifically, the motion quantity can be linear velocity, angular velocity, or other motion quantities that can be determined based on the displacement of the real-time point cloud cluster from the generation time of the historical point cloud to the generation time of the real-time point cloud.

[0045] Specifically, the clustering algorithm can be the DBSCAN clustering algorithm; based on the clustering algorithm, the real-time point cloud clusters obtained from real-time point cloud clustering and the historical point cloud clusters obtained from historical point cloud clustering are determined respectively, including: Based on the preset neighborhood radius and the preset number of sample points within a cluster, the real-time point cloud clusters obtained by real-time point cloud clustering and the historical point cloud clusters obtained by historical point cloud clustering are determined using the DBSCAN clustering algorithm.

[0046] In some embodiments of this application, the combinatorial optimization algorithm can be the Hungarian algorithm; determining the historical point cloud clusters that match the real-time point cloud clusters based on the combinatorial optimization algorithm, and using them as the matching results of the real-time point cloud clusters, includes: determining the historical point cloud clusters that match the real-time point cloud clusters based on the association cost formula of the Hungarian algorithm, and using them as the matching results of the real-time point cloud clusters, wherein the association cost formula includes the distance cost, shape matching cost, and volume matching cost of the two point cloud clusters.

[0047] Based on the above steps, distance cost, shape matching cost, and volume matching cost can be comprehensively considered to determine the matching result of real-time point cloud clusters. Understandably, the closer two point cloud clusters are, the lower the distance cost, and the better their shape and volume match; consequently, the shape matching cost and volume matching cost also decrease. The distance, shape, and volume of two point cloud clusters can all be determined by calculating the coordinates of each sampling point within the point cloud cluster.

[0048] Specifically, the association cost formula (1) is:

[0049] (1) in, Represents a real-time point cloud cluster. Represents historical point cloud clusters, This represents the association cost between two cloud clusters. This represents the distance cost between two cloud clusters. This represents the cost of shape matching between two cloud clusters. This represents the cost of matching the volume of two cloud clusters. , and These represent the weighting coefficients of each cost item.

[0050] In some embodiments of this application, step S202 of acquiring real-time pose data of the working equipment may include: acquiring pose data of key components, wherein the pose data of key components is determined based on the historical motion commands of the key components and the motion model of the working equipment.

[0051] Based on historical motion commands, the current pose of key components can be determined in the motion model, thereby determining the real-time pose data of the working equipment. In other words, the real-time pose data of the working equipment has been predetermined, and the current real-time motion commands of key components can be used to predict the pose data of subsequent key components in the motion model.

[0052] In some embodiments of this application, before acquiring the pose data of the key component, the 3D reconstruction method further includes: acquiring the motion command of the key component; determining the motion trajectory of the key component within a preset time period based on the motion command and the motion model of the operating equipment, wherein the motion trajectory includes the predicted pose data of the key component at multiple different timestamps; acquiring the pose data of the key component includes: selecting the predicted pose data corresponding to the first timestamp of the real-time pose data or the second timestamp that is the same as the first timestamp of the real-time point cloud as the pose data of the key component in the motion trajectory.

[0053] Specifically, as an example, the working equipment can be a crane, and the key component can be the crane's hook. The motion model of the working equipment can represent the poses of different crane components, such as the turntable, the luffing motor trolley, and the hook. The turntable's pose can be determined based on data from a slewing angle encoder, and the position of the luffing motor trolley and the hook height can also be acquired based on sensors. A local Cartesian coordinate system of the crane equipment can be established in the motion model of the working equipment as the coordinate system of the working equipment. The poses of the key components in the motion model can serve as initial values ​​and motion constraints for the key component's motion, and combined with motion commands, the motion trajectory of the key components can be predicted. For example, the hook height can serve as a motion constraint for the hook, and combined with the hook's motion commands, the swing trajectory of the hook can be predicted.

[0054] like Figure 4 As shown in some embodiments of this application, in order to more accurately determine the region of interest of key components, an image acquisition device may also be provided on the work equipment. The three-dimensional reconstruction method for work equipment provided in the embodiments of this application may further include: S402. Acquire real-time images captured by the image acquisition device; S404. Determine the relative positions of the measuring device and the image acquisition device in the coordinate system of the working equipment based on real-time pose data; S406. Determine the position of the object mask in the real-time image in the real-time point cloud based on the relative position, and determine the object point cloud cluster in the real-time point cloud based on the object mask.

[0055] Specifically, the image acquisition device can be, for example, a monocular or binocular camera. The object mask can be implemented based on a visual recognition model such as the YOLO model, and the object mask can be used to identify objects such as people, vehicles, crane hooks, and hoisted loads.

[0056] Step S206, which involves determining the region of interest (ROI) of the key components in the real-time point cloud based on their pose data and real-time position in the real-time pose data, and filtering out artifact points in the real-time point cloud based on a dynamic point cloud filtering algorithm, may include: Based on the pose data and real-time position of key components in the real-time pose data, S408 determines the region of interest of key components in the object point cloud cluster, and filters out artifact points in the object point cloud cluster based on the dynamic point cloud filtering algorithm.

[0057] Based on the above steps, the object point cloud clusters can be pre-determined using real-time images acquired by the image acquisition device and the object masks therein, thus narrowing down the range of the region of interest. The accurate region of interest can then be obtained based on the dual verification of the object mask and the pose data of key components.

[0058] In some embodiments of this application, the working device includes a motion node, the motion node is equipped with a pose sensor, and the real-time pose data includes the real-time node pose data of the motion node. Step S202, acquiring the real-time pose data of the working equipment, may include: acquiring the node pose of the motion node through a pose sensor; determining the real-time position of the measuring device in the coordinate system of the working equipment based on the real-time pose data includes: determining the real-time position of the measuring device in the coordinate system of the working equipment based on the motion model of the working equipment according to the real-time node pose data, wherein the motion model includes the positional relationship between the measuring device and the motion node, and the coordinate system of the working equipment is set based on the motion node.

[0059] Specifically, the working equipment is, for example, a crane, and the motion nodes include the crane slewing table and the luffing motor trolley on the boom. The measuring device is located on the luffing motor trolley, and the pose sensors include: a slewing angle encoder for the crane slewing table and a trolley pose sensor for the luffing motor trolley. Obtaining the node pose of the motion nodes through the pose sensors includes: obtaining the slewing angle of the crane slewing table through the slewing angle encoder; obtaining the trolley pose of the luffing motor trolley through the trolley pose sensor; determining the real-time position of the measuring device in the working equipment coordinate system based on the real-time node pose data in the motion model of the working equipment includes: determining the real-time position of the measuring device relative to the crane slewing table in the motion model based on the slewing angle and the trolley pose as the real-time position of the measuring device in the working equipment coordinate system.

[0060] In summary, taking a crane as the operating equipment and a lidar as the measuring device, the 3D reconstruction method for operating equipment provided in this application can utilize the crane's motion parameters to perform motion compensation on the original lidar point cloud, eliminating the influence of sensor pose changes caused by the crane's own rotation and amplitude variation, and significantly reducing point cloud distortion caused by its own motion. Furthermore, it combines visual semantic information (such as target detection and segmentation) to perform preliminary dynamic target segmentation on the compensated point cloud, identifying potential moving objects (vehicles, personnel), obtaining a real-time dynamic visual mask, and performing multimodal fusion real-time obstacle detection between the visual mask area and the corresponding point cloud area. Based on the segmented dynamic target's historical position and velocity information, combined with the crane hook's motion trajectory model and prior scene knowledge, the motion trajectory of dynamic obstacles is predicted in the near future. According to the predicted trajectory, a region of interest (ROI) with adaptive shape and size is dynamically generated around the predicted path, and this ROI is updated in real-time with the target's movement. The predicted adaptive ROI is used as the core processing area, where computational resources are concentrated, and optimized dynamic point cloud filtering algorithms (such as those combining motion compensation residuals and visual tracking information) are applied to accurately remove dynamic obstacles and their resulting "momentary" point clouds.

[0061] This application also provides a three-dimensional reconstruction apparatus for a work equipment, including: a processor configured to retrieve instructions from memory and, when executing the instructions, to implement the three-dimensional reconstruction method for a work equipment according to the above embodiments.

[0062] This application also provides a working device, including: a working device body; a three-dimensional reconstruction device for the working device provided in the above embodiments; a measuring device disposed on the working device body; and a pose sensor disposed on the working device body, the pose sensor being used to acquire real-time pose data of the working device.

[0063] In some embodiments of this application, the working equipment is a crane. The main body of the working equipment includes: a slewing support structure, which rotates via the crane's turntable; a boom connected to the slewing support structure; a hook, a key component, which is movably connected to the boom via a luffing motor trolley; a measuring device located on the luffing motor trolley; and a position sensor including: a slewing angle encoder located on the crane's turntable and a trolley position sensor located on the luffing motor trolley.

[0064] This application also provides a machine-readable storage medium storing instructions that cause a machine to execute a three-dimensional reconstruction method for a working device according to an embodiment of this application.

[0065] The steps in the flowchart of the three-dimensional reconstruction method for working equipment in this application embodiment are shown sequentially as indicated by the arrows. However, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0066] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements a method. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0067] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0068] In one embodiment, the three-dimensional reconstruction apparatus for working equipment provided in this application can be implemented as a computer program, which can be implemented in the form of, for example... Figure 5The computer device shown operates on this device. The computer device's memory can store various program modules that constitute the three-dimensional reconstruction apparatus for the work equipment. The computer program, composed of the various program modules, causes the processor to execute the steps in the three-dimensional reconstruction methods for the work equipment described in the various embodiments of this application.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0074] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0075] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0076] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0077] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for three-dimensional reconstruction of working equipment, characterized in that, The operating equipment is equipped with a measuring device; the three-dimensional reconstruction method includes: Acquire real-time pose data of the operating equipment and real-time point cloud data collected by the measuring device; The real-time position of the measuring device in the coordinate system of the working equipment is determined based on the real-time pose data. Based on the pose data of key components in the real-time pose data and the real-time position, the region of interest of the key components is determined in the real-time point cloud, and artifact points in the real-time point cloud are filtered out based on a dynamic point cloud filtering algorithm to obtain a processed point cloud with artifacts removed for use in the 3D reconstruction of the working equipment.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, The step of determining the region of interest (ROI) of the key component in the real-time point cloud based on the pose data of the key component in the real-time pose data and the real-time position, and filtering out artifact points in the real-time point cloud based on a dynamic point cloud filtering algorithm to obtain a processed point cloud with artifacts removed for use in the 3D reconstruction of the working equipment includes: Based on the pose data of key components in the real-time pose data and the real-time position, the region of interest of the key components is determined in the real-time point cloud, and a point cloud cluster within the region of interest is obtained. The algorithm uses dynamic point cloud filtering to filter out artifact points in the point cloud clusters within the region of interest.

3. The three-dimensional reconstruction method according to claim 1, characterized in that, The artifact points include discrete points, and the filtering of artifact points in the real-time point cloud based on the dynamic point cloud filtering algorithm includes: If the local density of any sampling point in the real-time point cloud is less than a preset local density threshold, the sampling point is filtered out from the real-time point cloud.

4. The three-dimensional reconstruction method according to claim 1, characterized in that, The artifact points include motion anomaly points, and the dynamic point cloud filtering algorithm includes a temporal consistency check algorithm; The method for filtering artifact points in the real-time point cloud based on the dynamic point cloud filtering algorithm includes: The motion anomaly points in the real-time point cloud are filtered out according to the temporal consistency test algorithm.

5. The three-dimensional reconstruction method according to claim 4, characterized in that, The step of filtering out motion anomalies in the real-time point cloud according to the temporal consistency check algorithm includes: Acquire historical point clouds, wherein the acquisition time of the historical point clouds is earlier than that of the real-time point clouds; The real-time point cloud clusters obtained by clustering the real-time point cloud and the historical point cloud clusters obtained by clustering the historical point cloud are determined based on the clustering algorithm. The historical point cloud clusters that match the real-time point cloud clusters are determined based on the combinatorial optimization algorithm and are used as the matching results of the real-time point cloud clusters. The motion of the real-time point cloud cluster from the generation time of the historical point cloud to the generation time of the real-time point cloud is determined based on the matching result. If the amount of motion exceeds the preset range, the real-time point cloud cluster is identified as an abnormal point of motion, and the real-time point cloud cluster is filtered out from the real-time point cloud.

6. The three-dimensional reconstruction method according to claim 5, characterized in that, The combined optimization algorithm is the Hungarian algorithm; the historical point cloud clusters that match the real-time point cloud clusters determined based on the combined optimization algorithm, as the matching results of the real-time point cloud clusters, include: The historical point cloud clusters that match the real-time point cloud clusters are determined based on the association cost formula of the Hungarian algorithm, and are used as the matching results of the real-time point cloud clusters. The association cost formula includes the distance cost, shape matching cost and volume matching cost of the two point cloud clusters.

7. The three-dimensional reconstruction method according to claim 6, characterized in that, The associated cost formula (1) is: ； （1） in, This refers to the real-time point cloud cluster. This refers to the historical point cloud cluster. This represents the association cost between two cloud clusters. This represents the distance cost between two cloud clusters. This represents the cost of shape matching between two cloud clusters. This represents the cost of matching the volume of two cloud clusters. , and These represent the weighting coefficients of each cost item.

8. The three-dimensional reconstruction method according to claim 1, characterized in that, The acquisition of the real-time pose data of the operating equipment includes: The pose data of the key component is acquired, wherein the pose data of the key component is determined based on the historical motion commands of the key component and the motion model of the working equipment.

9. The three-dimensional reconstruction method according to claim 8, characterized in that, Before acquiring the pose data of the key components, the 3D reconstruction method further includes: Obtain the motion commands of the key components; The motion trajectory of the key component within a preset time period is determined based on the motion command and the motion model of the working equipment. The motion trajectory includes the predicted pose data of the key component at multiple different timestamps. The acquisition of the pose data of the key components includes: In the motion trajectory, the predicted pose data corresponding to either the first timestamp of the real-time pose data or the second timestamp that is the same as the first timestamp of the real-time point cloud is selected as the pose data of the key component.

10. The three-dimensional reconstruction method according to claim 1, characterized in that, The operating equipment is also equipped with an image acquisition device, and the three-dimensional reconstruction method further includes: Acquire real-time images captured by the image acquisition device; The relative positions of the measuring device and the image acquisition device in the coordinate system of the working equipment are determined based on the real-time pose data. The position of the object mask in the real-time image in the real-time point cloud is determined based on the relative position, and the object point cloud cluster in the real-time point cloud is determined based on the object mask. The step of determining the region of interest (ROI) of the key component in the real-time point cloud based on the key component pose data in the real-time pose data and the real-time position, and filtering out artifact points in the real-time point cloud based on a dynamic point cloud filtering algorithm includes: Based on the pose data of key components in the real-time pose data and the real-time position, the region of interest of the key components is determined in the object point cloud cluster, and artifact points in the object point cloud cluster are filtered out based on a dynamic point cloud filtering algorithm.

11. The three-dimensional reconstruction method according to claim 1, characterized in that, The working device includes a motion node, and the motion node is equipped with a pose sensor. The real-time pose data includes the real-time node pose data of the motion node. The acquisition of the real-time pose data of the operating equipment includes: The node pose of the moving node is obtained through the pose sensor; Determining the real-time position of the measuring device in the coordinate system of the working equipment based on the real-time pose data includes: The real-time position of the measuring device in the coordinate system of the working equipment is determined based on the real-time node pose data in the motion model of the working equipment. The motion model includes the positional relationship between the measuring device and the motion node, and the coordinate system of the working equipment is set based on the motion node.

12. A three-dimensional reconstruction device for working equipment, characterized in that, include: The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement the three-dimensional reconstruction method for a working device according to any one of claims 1 to 11.

13. A working device, characterized in that, include: Main body of the operating equipment; The three-dimensional reconstruction apparatus for working equipment according to claim 12; The measuring device is located on the main body of the operating equipment; A pose sensor is installed on the main body of the working equipment, and the pose sensor is used to acquire the real-time pose data of the working equipment.

14. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform a three-dimensional reconstruction method for a working device according to any one of claims 1 to 11.

Images