Large mechanical radar vision fusion detection tracking method and system

Abstract

The application provides a large mechanical visual fusion detection tracking method and system, which comprises the following steps: processing a three-dimensional point in space into a two-dimensional pixel point in an image; designing a depth estimation network; in a training process, constructing a projection geometry loss function by using the geometric shape of a crane arm and the projection relationship of the crane arm in an image, taking the projection geometry loss function as a supervision signal to constrain the depth estimation network, inputting a monocular image of the crane arm into the depth estimation network, processing to obtain and output a predicted depth map; performing back projection according to the predicted depth map, and restoring the posture of the crane arm according to a three-dimensional point cloud to obtain the three-dimensional coordinates of the crane arm. The application solves the technical problems that the scale calibration cannot be completed by itself, the depth output exists overall deviation, the three-dimensional point cloud restoration of the crane arm is distorted, the spatial positioning deviation is large, and the posture solution stability is insufficient.

Description

Technical Field

[0001] This invention relates to the field of machine vision inspection and tracking, specifically to a laser vision fusion inspection and tracking method and system for large machinery. Background Technology

[0002] In applications involving large machinery such as wheeled cranes, tower cranes, and excavators, including automated operation, safety monitoring, and collision warning, acquiring the target's three-dimensional spatial position and attitude is one of the core technologies. In recent years, depth estimation methods based on monocular cameras have become a research hotspot in the field of visual measurement due to their advantages such as low hardware cost and simple deployment.

[0003] These methods typically utilize deep learning networks to regress pixel-level depth maps from single-frame images. The network is trained on large-scale datasets, learning to map image appearance features such as texture, occlusion, shadows, and perspective scaling to depth values. This existing method eliminates the need for stereo matching or multi-view acquisition, enabling visually plausible depth predictions across various scenarios. However, due to the lack of an absolute distance benchmark in a single image, the depth values ​​output by the network only maintain a relative order, exhibiting an unknown proportional deviation from the true physical scale, and cannot directly reconstruct metric-based 3D coordinates. Especially in distant, weakly textured regions, such as the far end of a crane arm or against a sky background, depth estimation is prone to issues like sparse point clouds, local drift, and depth jumps, resulting in poor 3D localization accuracy and unreliable pose recovery.

[0004] Existing monocular deep learning depth estimation methods rely solely on image feature inference and cannot perform scale calibration themselves, resulting in an overall offset in depth output. They are highly susceptible to depth drift, sparse point clouds at distant points, and structural distortion due to the influence of scene lighting, long-distance imaging, and weak texture interference. Conventional monocular solutions do not incorporate external measured geometric dimensions as constraints, nor do they combine the imaging projection relationship of large mechanical rigid structures for supervised correction of depth prediction. This fails to correct inherent biases in depth estimation, ultimately leading to distorted 3D point cloud reconstruction of the boom, large spatial positioning errors, and insufficient stability in attitude calculation, making it difficult to meet the engineering requirements for on-site safety monitoring, high-precision ranging, and attitude monitoring of large machinery.

[0005] The existing invention patent application document CN121572333A, entitled "A Welding Robot System and Intelligent Route Generation Method Based on Embodied Perception," describes a method that constructs a hierarchical feature processing system based on the channel and spatial attention mechanism of a 3D attention fusion submodule, the projection mapping model of a digital twin mapping submodule, and the attention weight calculation of a feature fusion submodule. This system strengthens key feature representation through the attention mechanism and achieves spatial alignment of point clouds and images, as well as dense 3D mapping of the welding area using digital twin technology. Core features are then extracted using the RANSAC algorithm and a convolutional neural network, and multi-dimensional information such as visual and arc data is fused. However, this existing technology utilizes fixed conditions in industrial scenarios and a twin-based model for data fusion to generate dense point clouds, which is unsuitable for modeling dense point clouds of real-time moving targets in complex open environments.

[0006] The existing invention patent application document CN121458863A, entitled "Three-Dimensional Semantic Scene Reconstruction Method, Apparatus, Electronic Device, and Storage Medium," describes a method that includes: using a Transformer architecture network to fuse features from multiple 2D images based on an attention mechanism, adaptively selecting the fused 2D features; finding the 2D projection point of each 3D voxel based on a camera projection mapping model, and calculating deformable cross-attention using only local 2D features near the projection point. This existing solution achieves coarse 3D static modeling of indoor scenes by capturing images from multiple angles, and cannot be applied to real-time 3D modeling, ranging, and tracking of moving targets. The existing method calls a pre-trained depth estimation model, whose output can only maintain the relative depth order and has an unknown proportional deviation from the real physical scale. Moreover, its loss function is a semantic classification loss rather than a geometric constraint, which cannot eliminate depth drift. At the same time, the existing technology is geared towards indoor scenes, such as working distances of 5-15 meters. For distant targets with weak metal textures, such as cranes, or against a sky background, the pre-trained model will exhibit depth jumps, sparse point clouds, and structural distortions. In addition, the existing technology uses discrete voxel grid representation, which cannot meet the requirements of high-precision pose calculation. Furthermore, the existing technology relies on multi-view cumulative completion of voxel space, and for occluded areas, it is based on semantic prior guesses and cannot recover the precise geometric position of the occluded parts.

[0007] In summary, existing technologies suffer from several technical problems, including the inability to perform scale calibration, overall offset in depth output, distortion in the reconstruction of the 3D point cloud of the boom, large spatial positioning deviation, and insufficient stability in attitude calculation. Summary of the Invention

[0008] The technical problem to be solved by this invention is: how to solve the technical problems in the prior art that it cannot complete scale calibration on its own, that there is an overall offset in depth output, that the 3D point cloud restoration of the boom is distorted, that the spatial positioning deviation is large, and that the attitude calculation stability is insufficient.

[0009] This invention solves the above-mentioned technical problems by employing the following technical solution: A large-scale mechanical laser vision fusion detection and tracking method includes: S1. Process three-dimensional points in space into two-dimensional pixels in the image; S2. Design a depth estimation network. During training, use the geometry of the crane boom and its projection relationship in the image to construct a projection geometry loss function as a supervision signal to constrain the depth estimation network. Input the monocular image of the crane boom into the depth estimation network, process it, and output the predicted depth map. S3. Perform back projection based on the predicted depth map, obtain the boom attitude based on the 3D point cloud, and calculate the 3D coordinates of the boom.

[0010] This invention combines projection geometric constraints with monocular depth estimation for joint optimization. It utilizes the physical length of a large mechanical crane boom obtained from on-site LiDAR measurements as a dimensional benchmark. By projecting this physical length onto the image plane, a geometric mapping constraint between the physical world and the image plane is constructed. This constraint corrects the depth information output by the initial monocular depth estimation network, optimizing depth prediction accuracy and 3D point cloud quality. The 3D point cloud is then obtained by backprojecting the depth map output by the converged depth estimation network, thereby recovering the crane boom's pitch angle, azimuth angle, and 3D spatial coordinates. Ultimately, this achieves high-precision 3D position calculation and attitude recovery. This technical solution relies on measured geometric dimensions provided by LiDAR and supervises monocular visual depth estimation through projection relationships, thereby improving perception robustness and measurement accuracy. It features convenient deployment, strong anti-interference capabilities, and high engineering practicality.

[0011] In a more specific technical solution, S2 involves detecting and extracting the mask of the crane boom region. Through segmentation networks, keypoint detection, and connected component analysis, the pixel region mask of the crane boom is extracted from the monocular image. ,in, arm Indicates the crane boom; Detect the pixel coordinates of the root hinge point A of the crane boom. and the pixel coordinates of the endpoint B and the skeleton point set of the crane arm. ,in N This represents the total number of pixels within the crane arm area; For the crane boom area i 1 pixel The predicted depth value is ;in, p Represents pixels, d Indicates depth, The depth function represents the predicted depth of the crane boom at its root hinge point A and end point B, denoted as [missing information]. , Using the camera intrinsic parameter matrix K Back projection yields 3D points in the camera coordinate system :

[0012] Using a set of three-dimensional points, the original predicted point cloud of the crane boom is constructed. .

[0013] During the training process of the depth estimation network of this invention, the geometric shape of the crane arm and its projection relationship in the image are used to construct a projection geometry loss function, which is used as a supervision signal to constrain the depth estimation network, thereby correcting the scale and drift of the depth estimation.

[0014] In a more specific technical solution, in S2, the original predicted point cloud is filtered to retain geometrically consistent point clouds; Statistically identify and remove outliers, calculate the average distance of each 3D point to its own neighborhood, and if the average distance exceeds the global average by two standard deviations, identify and delete outliers. To perform a projection consistency check, the pixel coordinates of the root hinge point A and the back-projected pixel coordinates of the end point B of the crane boom, obtained through back projection, are used. , Fit a straight line in space, remove points that deviate from the straight line, and obtain the effective point cloud. .

[0015] This invention optimizes monocular depth estimation by combining projection geometric constraints with projection geometric constraints. It obtains the actual length of the boom by LiDAR measurement and performs supervised correction on monocular depth estimation by combining projection geometric constraints. This effectively improves the problems of pure monocular depth prediction offset and depth drift, significantly optimizes the sparse point cloud, structural distortion and depth jump phenomenon in distant weak texture areas, and improves the accuracy of depth estimation and the quality of 3D point cloud reconstruction.

[0016] This invention performs three-stage filtering on the original predicted point cloud output by the depth estimation network: depth range filtering, statistical outlier removal, and projection consistency test, in order to suppress long-distance sparse noise.

[0017] This invention utilizes only a small amount of geometric priors and combines sparse point clouds to achieve semi-supervised monocular image depth estimation and dense point cloud generation. It achieves scale correction capability for relative depth estimation through a small amount of geometric priors and sparse point clouds, and combines visual recognition capability to achieve point cloud completion of occluded parts and moving target tracking.

[0018] In a more specific technical solution, during the filtering operation of the original predicted point cloud, depth values ​​are removed. 3D points smaller than the minimum working distance or greater than the maximum detection distance :

[0019] In the formula, and These are the minimum and maximum working depths, respectively.

[0020] In a more specific technical solution, during the operation of statistically analyzing and removing outliers, for each depth value... 3D points smaller than the minimum working distance or greater than the maximum detection distance Calculate its to k The average distance between the nearest neighbors :

[0021] in Point i of k Nearest neighbor index, Indicates the first j For each of the three-dimensional points corresponding to the nearest neighbor indices, calculate the average distance for all of them. mean and standard deviation : ,

[0022] Represents depth value The nearest neighbor index of a 3D point that is less than the minimum working distance or greater than the maximum detection distance is used to retain the applicable point set that meets the following conditions. :

[0023] in, This indicates the number of standard deviations exceeding the global mean.

[0024] In a more specific technical solution, the direction vector of the line is calculated. For applicable point sets Each point in Calculate the distance from the point to the line. AB straight-line distance :

[0025] Retention distance less than threshold The points:

[0026] in, Indicates a valid point.

[0027] In a more specific technical solution, S2 utilizes the physical length S of the crane boom measured on-site by LiDAR and rigid body constraints to construct a loss function, supervise the output of the depth estimation network, and determine the pixel coordinates of the root hinge point A of the crane boom. and the pixel coordinates of the endpoint B Using a pinhole projection model, the three-dimensional coordinates of the crane boom's root hinge point A and end point B in the camera coordinate system are determined. and , represented as back-projected pixel coordinates:

[0028] in, This indicates the depth of the root hinge point A. Indicates the depth of endpoint B. K Represents the camera intrinsic parameter matrix; The back-projected pixel coordinates of the root hinge point A and the end point B of the crane boom. and Substitute: (10), expanded, we get the following equation:

[0029] The projection constraints are constructed using the projection geometric loss function. As a supervision term for the depth estimation network:

[0030] in, This represents the predicted value of the physical length of the crane boom. This represents the depth estimate of endpoint B, obtained through depth smoothing loss. Encourage smooth variations in depth maps on the surface of the crane boom.

[0031] This invention combines projection geometry loss and depth smoothing loss to form a total loss function, which serves as a supervision term to achieve end-to-end optimization of the depth estimation network.

[0032] In a more specific technical solution, during the boom attitude recovery process in S3, three-dimensional points in the camera coordinate system are obtained through back projection operation; The unit vector is obtained by normalizing the direction vector of the crane boom. ; According to the unit vector Determine the pitch angle of the crane boom. Azimuth ; The three-dimensional spatial position of the crane boom is calculated based on preset extrinsic parameters.

[0033] This invention calculates the three-dimensional coordinates, pitch angle, and azimuth angle of the crane boom based on the corrected true depth, eliminating the adverse effects of depth error on attitude calculation and enhancing the robustness of attitude estimation. This invention does not rely on the inherent preset dimensions of the target, is easy to deploy, and is highly robust, meeting the high-precision and high-stability safety monitoring requirements of engineering sites.

[0034] In a more specific technical solution, the depth information is corrected by the root hinge point A and the end point B of the crane boom. and Back projection yields 3D points in the camera coordinate system: ,

[0035] in, K Represents the camera intrinsic parameter matrix. , This represents the pixel coordinates of the root hinge point A and the end point B of the crane boom.

[0036] In a more specific technical solution, a large-scale mechanical laser vision fusion detection and tracking system includes: The 3D point processing module is used to process spatial 3D points into 2D image pixels; The depth estimation network supervision module is used to design the depth estimation network. During the training process, the projection geometry loss function is constructed using the geometry of the crane boom and the projection relationship of the crane boom in the image. This function serves as a supervision signal to constrain the depth estimation network. The monocular image of the crane boom is input into the depth estimation network, processed, and the predicted depth map is output. The depth estimation network supervision module is connected to the model building module. The attitude recovery and coordinate processing module is used to back-project the predicted depth map to obtain the crane boom attitude based on the 3D point cloud and calculate the 3D coordinates of the crane boom. The attitude recovery and coordinate processing module is connected to the depth estimation network supervision module.

[0037] The present invention has the following advantages over the prior art: This invention combines projection geometric constraints with monocular depth estimation for joint optimization. It utilizes the physical length of a large mechanical crane boom obtained from on-site LiDAR measurements as a dimensional benchmark. By projecting this physical length onto the image plane, a geometric mapping constraint between the physical world and the image plane is constructed. This constraint corrects the depth information output by the initial monocular depth estimation network, optimizing depth prediction accuracy and 3D point cloud quality. The 3D point cloud is then obtained by backprojecting the depth map output by the converged depth estimation network, thereby recovering the crane boom's pitch angle, azimuth angle, and 3D spatial coordinates. Ultimately, this achieves high-precision 3D position calculation and attitude recovery. This technical solution relies on measured geometric dimensions provided by LiDAR and supervises monocular visual depth estimation through projection relationships, thereby improving perception robustness and measurement accuracy. It features convenient deployment, strong anti-interference capabilities, and high engineering practicality.

[0038] During the training process of the depth estimation network of this invention, the geometric shape of the crane arm and its projection relationship in the image are used to construct a projection geometry loss function, which is used as a supervision signal to constrain the depth estimation network, thereby correcting the scale and drift of the depth estimation.

[0039] This invention optimizes monocular depth estimation by combining projection geometric constraints with projection geometric constraints. It obtains the actual length of the boom by LiDAR measurement and performs supervised correction on monocular depth estimation by combining projection geometric constraints. This effectively improves the problems of pure monocular depth prediction offset and depth drift, significantly optimizes the sparse point cloud, structural distortion and depth jump phenomenon in distant weak texture areas, and improves the accuracy of depth estimation and the quality of 3D point cloud reconstruction.

[0040] This invention performs three-stage filtering on the original predicted point cloud output by the depth estimation network: depth range filtering, statistical outlier removal, and projection consistency test, in order to suppress long-distance sparse noise.

[0041] This invention utilizes only a small amount of geometric priors and combines sparse point clouds to achieve semi-supervised monocular image depth estimation and dense point cloud generation. It achieves scale correction capability for relative depth estimation through a small amount of geometric priors and sparse point clouds, and combines visual recognition capability to achieve point cloud completion of occluded parts and moving target tracking.

[0042] This invention combines projection geometry loss and depth smoothing loss to form a total loss function, which serves as a supervision term to achieve end-to-end optimization of the depth estimation network.

[0043] This invention calculates the three-dimensional coordinates, pitch angle, and azimuth angle of the crane boom based on the corrected true depth, eliminating the adverse effects of depth error on attitude calculation and enhancing the robustness of attitude estimation. This invention does not rely on the inherent preset dimensions of the target, is easy to deploy, and is highly robust, meeting the high-precision and high-stability safety monitoring requirements of engineering sites.

[0044] This invention solves the technical problems existing in the prior art, such as the inability to complete scale calibration on its own, overall offset in depth output, distortion in the reconstruction of the 3D point cloud of the boom, large spatial positioning deviation, and insufficient stability of attitude calculation. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the basic steps of a large-scale mechanical laser vision fusion detection and tracking method according to Embodiment 1 of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Example 1 like Figure 1 As shown, a laser vision fusion detection and tracking method for large machinery includes the following basic steps: S1. Process three-dimensional points in space into two-dimensional pixels in the image; In this embodiment, a camera pinhole projection mapping model is constructed; In this embodiment, the object's imaging point in the camera Equivalent to three-dimensional points in space ,express w Three-dimensional space. The camera pinhole projection mapping model from a three-dimensional point in space to a two-dimensional pixel in the image is as follows, obtained by sequentially transforming the world coordinate system through rigid body transformation and perspective projection to the pixel coordinate system: (1) in, For the camera intrinsic parameter matrix, Represents focal length. Principal point coordinates As the tilt factor, For the camera extrinsic matrix, 3 A rotation matrix of 3, t It is a translation matrix. These are the three-dimensional coordinates in the camera coordinate system. c It refers to a camera.

[0048] S2. Design a depth estimation network. During training, use the geometry of the crane boom and its projection relationship in the image to construct a projection geometry loss function as a supervision signal to constrain the depth estimation network. Input the monocular image of the crane boom into the depth estimation network, process it, and output the predicted depth map. In this embodiment, depth estimation network supervision is performed based on projection geometric constraints; In this embodiment, a depth estimation network is designed. The input of the network is a monocular image, and the output is a pixel-level predicted depth map. During training, a projection geometry loss function is constructed using the geometry of the crane boom and its projection relationship in the image. This function serves as a supervisory signal to constrain the network, thereby correcting the scale and drift of the depth estimation.

[0049] The crane boom region is detected and its mask extracted. Through segmentation networks, keypoint detection, and connected component analysis, the pixel region mask of the crane boom is extracted from the monocular image. ,in, arm Indicates the crane boom; In this embodiment, during the detection and mask extraction of the crane boom region, an instantiated segmentation network, such as Mask R-CNN, YOLACT, or keypoint detection and connected component analysis, is used to extract the pixel region mask of the crane boom from the input monocular image. .

[0050] Detect the pixel coordinates of the root hinge point A of the crane boom. and the pixel coordinates of the endpoint B and the skeleton point set of the crane arm. ,in N This represents the total number of pixels within the crane arm area; In this embodiment, a depth estimation network with an encoder-decoder structure is used. For example: DPT, MiDaS, network parameters are denoted as Input a monocular image. I Then, the network outputs a predicted depth map. .

[0051] For the crane boom area i 1 pixel The predicted depth value is ;in, p Represents pixels, d Indicates depth, The depth function represents the predicted depth of the crane boom at its root hinge point A and end point B, denoted as [missing information]. , Using the camera intrinsic parameter matrix K Back projection yields 3D points in the camera coordinate system : (2) Using a set of three-dimensional points, the original predicted point cloud of the crane boom is constructed. .

[0052] Since depth estimation networks may produce outliers in sparsely textured or occluded regions, the original predicted point cloud needs to be filtered to preserve geometrically consistent point clouds. The filtering operation includes: In the filtering operation of the original predicted point cloud, depth values ​​are removed. 3D points smaller than the minimum working distance or greater than the maximum detection distance : (3) in and These are the minimum and maximum working depths, respectively.

[0053] Statistically identify and remove outliers, calculate the average distance of each 3D point to its own neighborhood, and if the average distance exceeds the global average by two standard deviations, identify and delete outliers. In the operation of statistically analyzing and removing outliers, for each depth value 3D points smaller than the minimum working distance or greater than the maximum detection distance Calculate its to k The average distance between the nearest neighbors : (4) in Point i of k Nearest neighbor index, Indicates the first j For each of the three-dimensional points corresponding to the nearest neighbor indices, calculate the average distance for all of them. mean and standard deviation : , (5) Represents depth value The nearest neighbor index of a 3D point that is less than the minimum working distance or greater than the maximum detection distance is used to retain the applicable point set that meets the following conditions. : (6) in This indicates the removal of outliers that exceed the global average by two standard deviations.

[0054] To perform a projection consistency check, the pixel coordinates of the root hinge point A and the back-projected pixel coordinates of the end point B of the crane boom, obtained through back projection, are used. , Fit a straight line in space, remove points that deviate from the straight line, and obtain the effective point cloud. .

[0055] Calculate the direction vector of the line .for Each point in For applicable point sets Each point in Calculate the distance from the point to the line. AB straight-line distance : (7) Retention distance less than threshold The points: (8) in, Indicates the effective point, threshold Select based on the actual work scenario.

[0056] Finally, the effective point cloud is obtained after the above filtering. .

[0057] Using the physical length S of the crane boom measured on-site by LiDAR and rigid body constraints, a loss function is constructed to supervise the output of the depth estimation network. The output is then determined based on the pixel coordinates of the root hinge point A of the crane boom. and the pixel coordinates of the endpoint B Using a pinhole projection model, the three-dimensional coordinates of the crane boom's root hinge point A and end point B in the camera coordinate system are determined. and , represented as back-projected pixel coordinates: (9) in, This indicates the depth of the root hinge point A. Indicates the depth of endpoint B. K Represents the camera intrinsic parameter matrix; The Euclidean distance between the crane boom's root hinge point A and end point B in the camera coordinate system is used as the predicted value for the crane boom's length. : (10) Will and Substituting into equation (10) above, we obtain the following equation: (11) To achieve end-to-end optimization in deep learning, the projection constraints are constructed as a projection geometric loss function. As a supervision term for the depth estimation network: (12) in, This represents an estimated value of the physical length of the crane boom. This represents the depth estimate of endpoint B, obtained through depth smoothing loss. Encourage smooth changes in depth maps on the boom surface.

[0058] Then use deep smoothing loss Encourage smooth transitions in the depth map on the crane boom surface to avoid jagged edges or isolated noise points.

[0059] (13) in, It is the total number of pixels covered by the pixel area mask of the crane arm. u , v ) represents pixel coordinates, D This represents the depth function.

[0060] The final network loss of this invention for: (14) S3. Perform back projection based on the predicted depth map, obtain the boom attitude based on the 3D point cloud, and calculate the 3D coordinates of the boom.

[0061] In this embodiment, the depth information is corrected by the root hinge point A and the end point B of the crane boom. and Back projection yields three-dimensional points in the camera coordinate system; Specifically, for the crane boom attitude recovery calculation, it first starts with the corrected depth information. and Back projection yields 3D points in the camera coordinate system: , (15) in, K This represents the camera intrinsic parameter matrix. , Represents the pixel coordinates of the root hinge point A and the end point B of the crane boom; Crane boom direction vector Then it is: (16) , , Indicate the x, y, and y coordinates of the root hinge point A after correction. , , This represents the horizontal, vertical, and axial coordinates of the corrected end point B.

[0062] The unit vector is obtained by normalizing the direction vector of the crane boom. : (17) According to the unit vector Determine the crane boom pitch angle That is, the angle relative to the horizontal plane is: (18) Crane boom azimuth That is, the projection in the horizontal plane and the camera x The included angle of the axis is: (19) Regarding the three-dimensional spatial position of the boom in the physical world and It can be calculated from external parameters. , (20).

[0063] In summary, this invention combines projection geometric constraints with monocular depth estimation for joint optimization. It utilizes the physical length of a large mechanical crane boom obtained from on-site measurements using LiDAR as a scale benchmark. Through the projection relationship of this physical length onto the image plane, a geometric mapping constraint between the physical world and the image plane is constructed. This constraint corrects the depth information output by the initial monocular depth estimation network, optimizing depth prediction accuracy and 3D point cloud quality. The 3D point cloud is then obtained by backprojecting the depth map output by the converged depth estimation network, thereby recovering the crane boom's pitch angle, azimuth angle, and 3D spatial coordinates. Ultimately, this achieves high-precision 3D position calculation and attitude recovery. This technical solution relies on measured geometric dimensions provided by LiDAR and supervises monocular visual depth estimation through projection relationships, thereby improving perception robustness and measurement accuracy. It features convenient deployment, strong anti-interference capabilities, and high engineering practicality.

[0064] During the training process of the depth estimation network of this invention, the geometric shape of the crane arm and its projection relationship in the image are used to construct a projection geometry loss function, which is used as a supervision signal to constrain the depth estimation network, thereby correcting the scale and drift of the depth estimation.

[0065] This invention optimizes monocular depth estimation by combining projection geometric constraints with projection geometric constraints. It obtains the actual length of the boom by LiDAR measurement and performs supervised correction on monocular depth estimation by combining projection geometric constraints. This effectively improves the problems of pure monocular depth prediction offset and depth drift, significantly optimizes the sparse point cloud, structural distortion and depth jump phenomenon in distant weak texture areas, and improves the accuracy of depth estimation and the quality of 3D point cloud reconstruction.

[0066] This invention performs three-stage filtering on the original predicted point cloud output by the depth estimation network: depth range filtering, statistical outlier removal, and projection consistency test, in order to suppress long-distance sparse noise.

[0067] This invention utilizes only a small amount of geometric priors and combines sparse point clouds to achieve semi-supervised monocular image depth estimation and dense point cloud generation. It achieves scale correction capability for relative depth estimation through a small amount of geometric priors and sparse point clouds, and combines visual recognition capability to achieve point cloud completion of occluded parts and moving target tracking.

[0068] This invention combines projection geometry loss and depth smoothing loss to form a total loss function, which serves as a supervision term to achieve end-to-end optimization of the depth estimation network.

[0069] This invention calculates the three-dimensional coordinates, pitch angle, and azimuth angle of the crane boom based on the corrected true depth, eliminating the adverse effects of depth error on attitude calculation and enhancing the robustness of attitude estimation. This invention does not rely on the inherent preset dimensions of the target, is easy to deploy, and is highly robust, meeting the high-precision and high-stability safety monitoring requirements of engineering sites.

[0070] This invention solves the technical problems existing in the prior art, such as the inability to complete scale calibration on its own, overall offset in depth output, distortion in the reconstruction of the 3D point cloud of the boom, large spatial positioning deviation, and insufficient stability of attitude calculation.

[0071] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A laser vision fusion detection and tracking method for large-scale machinery, characterized in that, The method includes: S1. Process three-dimensional points in space into two-dimensional pixels in the image; S2. Design a depth estimation network. During training, use the geometry of the crane boom and its projection relationship in the image to construct a projection geometry loss function as a supervision signal to constrain the depth estimation network. Input the monocular image of the crane boom into the depth estimation network, process it, and output the predicted depth map. S3. Perform back projection based on the predicted depth map, obtain the boom attitude based on the 3D point cloud, and calculate the 3D coordinates of the boom.

2. The laser vision fusion detection and tracking method for large machinery according to claim 1, characterized in that, In step S2, the crane boom region is detected and its mask extracted. Through segmentation networks, keypoint detection, and connected component analysis, the pixel region mask of the crane boom is extracted from the monocular image. ,in, arm Indicates the crane boom; Detect the pixel coordinates of the root hinge point A of the crane boom. and the pixel coordinates of the endpoint B and the skeleton point set of the crane arm. ,in N This represents the total number of pixels within the crane arm area; For the crane boom area i 1 pixel The predicted depth value is ;in, p Represents pixels, d Indicates depth, The depth function represents the predicted depth of the crane boom at its root hinge point A and end point B, denoted as [missing information]. , Using the camera intrinsic parameter matrix K Back projection yields 3D points in the camera coordinate system : Using the set of three-dimensional points, the original predicted point cloud of the crane boom is constructed. .

3. The laser vision fusion detection and tracking method for large-scale machinery according to claim 1, characterized in that, In step S2, the original predicted point cloud is filtered to retain geometrically consistent point clouds. Outliers are identified and removed. The average distance from each 3D point to its own neighborhood is calculated. If the average distance exceeds the global average by two standard deviations, outliers are identified and deleted. To perform a projection consistency check, the pixel coordinates of the root hinge point A and the back-projected pixel coordinates of the end point B of the crane boom, obtained through back projection, are used. , Fit a spatial straight line, and remove points that deviate from the spatial straight line to obtain an effective point cloud. .

4. The laser vision fusion detection and tracking method for large machinery according to claim 3, characterized in that, In the filtering operation of the original predicted point cloud, depth values ​​are removed. 3D points smaller than the minimum working distance or greater than the maximum detection distance : In the formula, and These are the minimum and maximum working depths, respectively.

5. The laser vision fusion detection and tracking method for large machinery according to claim 3, characterized in that, In the operation of statistically analyzing and removing outliers, for each depth value 3D points smaller than the minimum working distance or greater than the maximum detection distance Calculate its to k The average distance between the nearest neighbors : in Point i of k Nearest neighbor index, Indicates the first j For each of the three-dimensional points corresponding to the nearest neighbor indices, calculate the average distance for all of them. mean and standard deviation : ， Represents depth value The nearest neighbor index of a 3D point that is less than the minimum working distance or greater than the maximum detection distance is used to retain the applicable point set that meets the following conditions. : in, This represents the number of standard deviations exceeding the global mean.

6. The laser vision fusion detection and tracking method for large machinery according to claim 3, characterized in that, Calculate the direction vector of the line For applicable point sets Each point in Calculate the distance from the point to the line. AB straight-line distance : Retention distance less than threshold The point: in, Indicates the valid point.

7. The laser vision fusion detection and tracking method for large machinery according to claim 1, characterized in that, In step S2, a loss function is constructed using the physical length S of the crane boom measured on-site by the lidar and rigid body constraints. This function is then used to supervise the output of the depth estimation network, based on the pixel coordinates of the root hinge point A of the crane boom. and the pixel coordinates of the endpoint B Using a pinhole projection model, the three-dimensional coordinates of the crane boom's root hinge point A and end point B in the camera coordinate system are determined. and , represented as back-projected pixel coordinates: in, Indicates the depth of the root hinge point A. Indicates the depth of endpoint B. K Represents the camera intrinsic parameter matrix; The back-projected pixel coordinates of the root hinge point A and the end point B of the crane boom. and Substitute: (10), expanded, we get the following equation: The projection constraints are constructed using the projection geometric loss function. As a supervision term for the depth estimation network: in, This represents the predicted value of the physical length of the crane boom. This represents the depth estimate of endpoint B, obtained through depth smoothing loss. Encourage smooth variations in depth maps on the surface of the crane boom.

8. The laser vision fusion detection and tracking method for large machinery according to claim 1, characterized in that, During the boom attitude recovery process in S3, three-dimensional points in the camera coordinate system are obtained through back projection operation; The unit vector is obtained by normalizing the direction vector of the crane boom. ; According to the unit vector Determine the pitch angle of the crane boom. Azimuth ; The three-dimensional spatial position of the crane boom is calculated based on preset extrinsic parameters.

9. The laser vision fusion detection and tracking method for large-scale machinery according to claim 8, characterized in that, Depth information corrected by the root hinge point A and end point B of the crane boom. and Back projection yields 3D points in the camera coordinate system: ， in, K Represents the camera intrinsic parameter matrix. , This represents the pixel coordinates of the root hinge point A and the end point B of the crane boom.

10. A large-scale mechanical laser vision fusion detection and tracking system, characterized in that, The system includes: The 3D point processing module is used to process spatial 3D points into 2D image pixels; The depth estimation network supervision module is used to design the depth estimation network. During the training process, the projection geometry loss function is constructed using the geometry of the crane arm and the projection relationship of the crane arm in the image. This function serves as a supervision signal to constrain the depth estimation network. The monocular image of the crane arm is input into the depth estimation network, processed, and a predicted depth map is output. The depth estimation network supervision module is connected to the 3D point processing module. The attitude recovery and coordinate processing module is used to back-project the predicted depth map to obtain the crane boom attitude based on the 3D point cloud and to calculate the 3D coordinates of the crane boom. The attitude recovery and coordinate processing module is connected to the depth estimation network supervision module.

Images