Laser radar and camera fused ship loader measuring system and method

Abstract

The invention belongs to the technical field of port and wharf equipment, and particularly discloses a laser radar and camera fused ship loader measuring system and method. The system comprises two vision and distance measurement assemblies, the two vision and distance measurement assemblies are symmetrically arranged on the two sides of a ship loader chute tube, and each vision and distance measurement assembly comprises a wide-angle camera, a long-distance camera and a short-distance laser radar; wherein the wide-angle camera is used for 180-degree view angle coverage acquisition; the long-distance camera is used for improving the resolution ratio of the bow and the stern; the short-distance laser radar is used for acquiring short-distance high-density point cloud data and complementing information loss of a shielding area around the ship loader chute. According to the invention, full-coverage measurement of the three-dimensional pose of the ship body can be realized under the requirement of low cost.

Description

Technical Field

[0001] This application belongs to the technical field of port terminal equipment. More specifically, it relates to a ship unloader measurement system and method that integrates lidar and cameras. Background Art

[0002] As the core equipment for loading bulk and general cargo at ports, the accuracy and efficiency of the ship unloader's operation directly affect the port throughput and operating costs.

[0003] Traditional ship unloaders rely on manual operation and are difficult to meet the high-precision and high-dynamic requirements of modern ports. Early vision measurement technology mainly used cameras for simple material monitoring, which could only assist manual judgment of the material accumulation state and could not achieve precise perception of the ship's pose. With the introduction of lidar technology, ship unloaders have gradually gained the ability of environmental perception and ship positioning. However, limited by cost and dynamic adaptability, the core problems such as complex occlusion and high-precision measurement at long distances have not been completely solved.

[0004] Therefore, it is necessary to provide a ship unloader measurement system with low cost and no blind spots. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the purpose of this application is to provide a ship unloader measurement system and method that integrates lidar and cameras, which can achieve full coverage measurement of the three-dimensional pose of the hull under the requirement of low cost.

[0006] To achieve the above purpose, in the first aspect, this application provides a ship unloader measurement system that integrates lidar and cameras, including two sets of vision and ranging components. The two sets of vision and ranging components are symmetrically arranged on both sides of the ship unloader chute. Each set of vision and ranging components includes a wide-angle camera, a long-distance camera, and a short-distance lidar. Among them, the wide-angle camera is used for 180° perspective coverage acquisition; the long-distance camera is used to improve the resolution of the bow and stern; the short-distance lidar is used to obtain high-density point cloud data at close range to make up for the missing information in the occlusion area around the ship unloader chute.

[0007] The beneficial effects of the present application are as follows: the ship loader measurement system integrating laser radar and camera provided in the present application has the following effects: (1) the hardware cost of the system can be greatly reduced by replacing the traditional high-cost multiple long-distance laser radars with a low-cost camera combination; (2) two short-distance laser radars and four camera units are used. Considering the fact that the part farther away from the wide-angle camera has a lower effective resolution in the imaging result when the perspective relationship exists, the present embodiment adds two long-distance cameras with a narrower field of view to supplement the resolution of the far end (bow and stern); at the same time, considering that the quality of the close-range hull scanning point cloud is affected by the camera resolution, the present application adds two groups of high-precision short-range laser radars to improve the resolution of the point cloud near the chute, thereby effectively solving the problems of occlusion blind spots and insufficient long-distance resolution existing in the traditional single laser radar, and realizing high-precision and full-coverage measurement of the three-dimensional position of the hull.

[0008] As a further preference, the vision and ranging component is fixed on a pan / tilt platform of the ship loader, the pan / tilt platform has an angle adjustment function of at least one degree of freedom, the rotation angular velocity is controlled within the range of 0.05° / s to 2° / s, and the direction of its rotation axis is consistent with the longitudinal direction of the hull.

[0009] As a further preference, the wide-angle camera adopts a wide-angle camera with a downward viewing angle of 180°, the telephoto camera adopts a telephoto camera with a lateral viewing angle of 70°, and the short-range laser radar adopts a short-range laser radar with a 360° all-round viewing angle.

[0010] In a second aspect, the present application provides a measurement method of a ship loader measurement system integrating a laser radar and a camera, comprising the following steps: S10, move the slide tube to the top of the empty ship, then turn on two short-range laser radars, two wide-angle cameras and two long-range cameras installed on the slide tube seat, and keep the long-range cameras facing the bow or stern of the ship in a pan-tilt linkage to perform the first attitude measurement of the ship; S20, switch to the automatic operation mode, the ship loader control system automatically controls the chute to move to a position close to the bow, and then moves to the stern at a constant speed; during the movement of the chute, whenever the position of the chute changes, the system will perform pan-tilt linkage and attitude measurement to ensure that complete information about the hull is obtained; after the scanning is completed, the scanning results of each part obtained by the close-range laser radar are spliced and integrated to obtain a three-dimensional point cloud model M1; S30, using two groups of four cameras to collect RGB images of the hull, using point and line features for matching, and using the SFM algorithm to obtain a 3D reconstructed hull point cloud model M2; S40, align M1 and M2; S50. During the actual process of loading, once there is a need to update the real-time status of the hull, obtain the updated hull model.

[0011] As a further preference, the attitude measurement is specifically: Establish the point feature matching relationship ( , ) and the line feature matching relationship ( , ) between the current RGB image and the previous frame of RGB image; where is the point feature of the previous frame of RGB image, is the point feature of the current RGB image, is the line feature of the previous frame of RGB image, is the line feature of the current RGB image; Construct the first loss function, and the first loss function includes the first point cloud alignment loss function, the first point reprojection loss function, and the first line reprojection loss function; Iteratively optimize the pose so that the value of the first loss function meets the requirements, and obtain the rotation R 1 and translation T 1 of the final reliable pose, R 1 = R · R 0, T 1 = T 0 - T ; where R is the estimated rotation, T is the estimated translation, R 0 and T 0 are the rotation and translation values in the initial pose of the previous frame respectively.

[0012] As a further preference, the calculation formula of the first point cloud alignment loss function is: where

[0013] The calculation formula of the first point reprojection loss function is:

[0014] The calculation formula of the first line reprojection loss function is:

[0015] The calculation formula of the first loss function is:

[0016] Wherein, is the number of points in the point cloud; is the i -th point in the previous frame of point cloud; is the estimated value of the i -th point in the current frame of point cloud; is the number of point features; is the position of the i -th point feature in the previous frame of point cloud; is the estimated value of the position of the i -th point feature in the current frame; is the number of line features; is the position of the i -th line feature in the previous frame of point cloud; is the estimated value of the position of the i -th line feature in the current frame.

[0017] As a further preference, the pan-tilt linkage is as follows: when the boom of the ship unloader rotates, the two pan-tilts need to be synchronized so that the long-distance camera always looks at the bow and stern of the ship.

[0018] As a further preference, the pan-tilt linkage is specifically as follows: When the boom of the ship unloader rotates, the two pan-tilts rotate accordingly to keep the long-distance camera always pointing in the directions of the bow and stern; when the pan-tilt rotates to a new angle, calibrate the rotation angle of the pan-tilt; Establish the point feature matching relationship ( , ) and the line feature matching relationship ( , ) between the current RGB image and the previous RGB image; wherein, is the point feature of the previous RGB image, is the point feature of the current RGB image, is the line feature of the previous RGB image, is the line feature of the current RGB image; Construct a second loss function, and the second loss function includes a second point cloud alignment loss function, a second point reprojection loss function, and a second line reprojection loss function; Iteratively optimize the pose so that the value of the second loss function meets the requirements, thereby calculating the final rotation angle of the pan-tilt R .

[0019] As a further preference, the calculation formula of the second point cloud alignment loss function is:

[0020] The second point reprojection loss function The calculation formula of

[0021] The second line re-projection loss function The calculation formula of

[0022] The second loss function The calculation formula of

[0023] In the formula, is the number of points in the point cloud; is the i th point in the previous frame of point cloud; is the estimated value of the i th point in the current frame of point cloud; is the number of point features; is the position of the i th point feature in the previous frame of point cloud; is the estimated value of the position of the i th point feature in the current frame; is the number of line features; is the position of the i th line feature in the previous frame of point cloud; is the estimated value of the position of the i th line feature in the current frame.

[0024] It can be understood that the beneficial effects of the above second aspect can be referred to the relevant descriptions in the above first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is the front installation view of the ship unloader measurement system with the fusion of lidar and camera provided by the embodiment of the present application; Figure 2 is the bottom installation view of the ship unloader measurement system with the fusion of lidar and camera provided by the embodiment of the present application; Figure 3 is the perspective projection view of the ship unloader measurement system with the fusion of lidar and camera provided by the embodiment of the present application; Figure 4 is the schematic diagram of the scanning range of each camera in the ship unloader measurement system with the fusion of lidar and camera provided by the embodiment of the present application; Figure 5 is the schematic diagram of the pan-tilt linkage provided by the embodiment of the present application.

[0026] In all the drawings, the same reference numerals are used to represent the same elements or structures, where: 1 is the chute seat, 2 is the chute, 3 is the pan-tilt head, 4 is the long-distance camera, 5 is the wide-angle camera, and 6 is the short-distance lidar. Specific embodiments

[0027] In order to make the objectives, technical solutions, and advantages of this application clearer, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0028] It should be understood that in the description of this application, the meaning of the term "several" is at least one, such as one, two, etc., unless otherwise specifically defined; the meaning of the term "multiple" is two or more, unless otherwise specifically defined; the terms "first" and "second" etc. are used to distinguish different objects rather than to describe the specific order of the objects; the term "and / or" includes any and all combinations of one or more of the related listed items.

[0029] In addition, the reference to "one embodiment" throughout this specification; the language such as "one embodiment", "one example", or the like means that the specific features, structures, or characteristics described in connection with that embodiment are included in at least one embodiment of this application. Therefore, the appearance of the phrase "in one embodiment;" throughout this specification and similar language may or may not all refer to the same embodiment.

[0030] Through research, it is found that the lidar technologies mainly used in traditional dynamic measurement of ship loaders include the mechanism-driven two-dimensional lidar solution, the horizontally installed long-distance lidar solution, and the multi-lidar data stitching solution.

[0031] Among them, the mechanism-driven two-dimensional lidar solution, that is, the method of scanning and stitching point clouds by the mechanism-driven two-dimensional lidar, scans through a two-dimensional lidar combined with a mechanical drive mechanism to obtain point cloud data at multiple angles, and stitches them into a complete hull model through a three-dimensional reconstruction algorithm.

[0032] This method has high flexibility, but its detection results are easily affected by the occlusion of objects such as chutes. At the same time, this method has a slow scanning speed and low resolution, and cannot meet the real-time requirements and precise positioning requirements of the dynamic operation of the ship loader. Moreover, this method requires the configuration of multiple high-precision two-dimensional lidars, mechanical rotation mechanisms, and highly reliable control systems, resulting in complex system integration, high equipment investment costs, and long maintenance cycles, which is not conducive to promotion and large-scale deployment.

[0033] The horizontal installation long-distance lidar solution scans the hull from above through a horizontal long-distance lidar and stitches the point cloud data, which has the advantages of large coverage and relatively simple installation. However, due to the occlusion of the shiploader structure such as the chute and the conveyor belt support, the data of key parts such as the bow and stern are missing, and the point cloud information of the material pile in the open hatch is incomplete due to the upper scanning angle, affecting the measurement accuracy; in addition, the scanning speed of a single lidar is limited and it is difficult to meet the dynamic operation requirements.

[0034] The multi-lidar data stitching solution deploys multiple lidars and uses a data fusion algorithm to stitch the point clouds collected by each lidar into a complete three-dimensional model, which is suitable for use in the presence of occlusion. However, in order to cover all angles and occlusion areas, multiple lidars need to be deployed (such as 6 LIVOX Horizons), which not only increases the procurement cost but also improves the installation and debugging difficulty.

[0035] In this context, the present application proposes a shiploader measurement system and method that combines lidar and cameras. By replacing the long-distance lidar with a low-cost camera and combining the short-distance lidar, wide-angle camera, and long-distance camera with pan-tilt linkage technology, high-precision and high-speed measurement of the three-dimensional pose of the hull can be achieved. Compared with the traditional lidar solution, the present application significantly reduces the hardware cost while breaking through technical bottlenecks such as occlusion area coverage, long-distance high-precision measurement, and dynamic real-time performance through a dynamic stitching algorithm and multi-sensor data fusion, providing a high-cost-effective and highly reliable solution for port loading operations.

[0036] As Figures 1 to 3 shown, the shiploader measurement system that combines lidar and cameras provided by the present application mainly includes two groups of vision and ranging components, which are symmetrically arranged on both sides of the shiploader chute. Each group of vision and ranging components includes a wide-angle camera 5, a long-distance camera 4, and a short-distance lidar 6.

[0037] Among them, the wide-angle camera 5 is used to collect data with a 180° view angle coverage; the long-distance camera 4 is used for fine texture observation at a long distance to improve the resolution of the far end (bow and stern); the short-distance lidar 6 is used to obtain high-density point cloud data at a short distance to make up for the missing information in the occlusion area around the shiploader chute 2.

[0038] In this embodiment, the wide-angle camera 5 can be a wide-angle camera with a 180° downward view angle, the long-distance camera 4 can be a long-distance camera with a 70° downward view angle, and the short-distance lidar 6 can be a short-distance lidar with a 360° omnidirectional view angle.

[0039] The ship unloader measurement system integrating lidar and camera provided in this embodiment has the following effects: (1) By replacing traditional high-cost multiple long-distance lidars with low-cost camera combinations, the hardware cost of the system can be significantly reduced; (2) With two short-distance lidars and four camera units, considering the characteristic that the effective resolution of the part farther from the wide-angle camera is lower in the imaging result due to the perspective relationship, two long-distance cameras with narrower field of view are added in this embodiment to supplement the resolution of the distal part (bow and stern); at the same time, considering that the quality of the scanned point cloud of the hull at close range is affected by the camera resolution, two groups of high-precision short-distance lidars are added in this application to improve the resolution of the point cloud near the chute, thus effectively solving problems such as occlusion blind spots and insufficient long-distance resolution existing in traditional single lidars, and achieving high-precision and full-coverage measurement of the three-dimensional pose of the hull.

[0040] The following will combine specific embodiments to elaborate in detail on the ship unloader measurement system integrating lidar and camera provided in this application and its measurement method.

[0041] Embodiment 1 Although the long-distance detection ability of the long-distance lidar can provide strong support for obtaining key information, it is usually costly, and there are certain limitations in the observation field of view of this lidar in the vertical direction, and its observation angle is relatively narrow. In contrast, the short-distance lidar has the significant advantage of low cost. Although its effective detection distance is short and the measurement range is limited to the surrounding area at close range, its angle of view is usually wide, and even a 360° omnidirectional measurement can be achieved. The organic combination of the two can achieve comprehensive information acquisition. However, considering the high price of the long-distance lidar, the hardware system of this embodiment takes the replacement of the long-distance lidar with a low-cost camera as the core, and adopts a multi-sensor (i.e., short-distance lidar 6, wide-angle camera 5 and long-distance camera 4) collaborative architecture. As shown in FIGS. 1 and 2, it includes two short-distance lidars 6 (360° omnidirectional view), two wide-angle cameras 5 (180° downward view) and two long-distance cameras 4 (70° lateral view), and is mounted on two ship unloader gimbals 3 with a ±30° rotation range. The sensors are symmetrically installed on both sides of the ship unloader chute 2 to form complementary coverage: the short-distance lidar 6 and the wide-angle camera 5 eliminate the blind spot under the chute 2, and the long-distance camera 4 is used to improve the resolution of the distal bow and stern. At the same time, the hardware cost is significantly lower than the traditional long-distance lidar solution, making its promotion in engineering significantly improved.

[0042] Specifically, as Figure 3As shown in the figure, a set of vision and ranging components are symmetrically arranged on both sides of the chute 2. Each side includes a wide-angle camera 5, a long-distance camera 4, and a short-distance lidar 6. The wide-angle camera 5 in each set of components is used for large-angle coverage acquisition, the long-distance camera 4 is used for long-distance fine texture observation, and the short-distance lidar 6 is responsible for obtaining short-distance high-density point cloud data to supplement the information missing in the occluded area around the chute of the ship loader. The above components are firmly fixed on the pan-tilt 3 through a stable connection structure, with one pan-tilt on each side, ensuring that the sensors always point towards the bow and stern directions during operation. The pan-tilt 3 needs to have an angle adjustment function with at least one degree of freedom, and the rotational angular velocity can be controlled within the range of 0.05° / s to 2° / s, and the direction of its rotation axis needs to be guaranteed to be consistent with the longitudinal direction of the hull, so that when the system performs hull scanning, the sensor orientation can be flexibly and accurately adjusted to meet the requirements under different working conditions.

[0043] Embodiment 2 When using the ship loader measurement system integrating lidar and camera provided in the above Embodiment 1 for industrial operations, the following steps are adopted: Step 1: Manual initialization: The chute 2 needs to be moved to directly above an empty ship, and then two short-distance lidars, two 180° wide-angle cameras, and two long-distance cameras installed on the chute seat 1 are turned on. The pan-tilt is linked to keep the long-distance camera facing the bow or stern for the first visual scanning and positioning of the hull.

[0044] Step 2: Switch to the automatic operation mode. The ship loader control system will automatically control the chute to move to a position close to the bow, and then move uniformly towards the stern direction to perform fine visual scanning of the hull. During the movement of the chute, whenever the position of the chute changes, the system will perform pan-tilt linkage and attitude measurement to ensure obtaining complete information of the hull. After the scanning is completed, the scanning results of each part obtained by the lidar are stitched and integrated to obtain an accurate and complete three-dimensional point cloud model M1. In this way, since the scanning viewpoints achieve full coverage and there are no visual dead angles, the constructed cabin model will not have void phenomena.

[0045] Step 3: Use a total of four cameras in two groups to collect RGB images of the hull, use point-line features for matching, and use the SFM algorithm to obtain the hull point cloud model M2 after three-dimensional reconstruction.

[0046] Step 4: Align M1 and M2.

[0047] Step 5: During the actual process of loading the ship, once there is a need to update the real-time state of the hull, the updated hull model can be obtained immediately to ensure accurately grasping the dynamic changes in the hull position and the real-time state changes in the material volume, providing a strong guarantee for the smooth progress of the ship loading operation.

[0048] In Embodiment 1, two close-range lidars and four groups of camera units are innovatively installed. Considering the characteristic that the effective resolution of the part farther from the wide-angle camera is lower in the imaging result in the presence of perspective relationship, two long-range cameras with narrower field of view are added in Embodiment 1 to complement the resolution of the distal part (bow and stern). At the same time, considering that the quality of the close-range hull scan point cloud is affected by the camera resolution, two groups of high-precision close-range lidars are added in Embodiment 1 to improve the resolution of the point cloud near the chute. As Figure 4 shown, the circular dotted line is the scanning range of the two lidars, the square dotted line is the scanning range of the two wide-angle cameras, and the trapezoidal dotted line is the scanning range of the long-range cameras. In this way, during the rotation of the boom of the ship loader, even with many variables, the scanning area can still completely cover the entire hull, and a high-quality point cloud model of the hull can be obtained at a low cost.

[0049] In Embodiment 2, the attitude measurement is specifically as follows: Step A: Establish the point feature matching relationship ( , ) and the line feature matching relationship ( , ) between the current RGB image and the previous frame of RGB image.

[0050] Among them, is the point feature of the previous frame of RGB image, is the point feature of the current RGB image, is the line feature of the previous frame of RGB image, is the line feature of the current RGB image.

[0051] Step B: Construct the first loss function, which includes the first point cloud alignment loss function, the first point reprojection loss function, and the first line reprojection loss function.

[0052] In this embodiment, the calculation formula of the first point cloud alignment loss function is:

[0053] The calculation formula of the first point reprojection loss function is:

[0054] The calculation formula of the first line reprojection loss function is:

[0055] The calculation formula of the first loss function is:

[0056] In the formula, is the number of points in the point cloud; is the i -th point in the previous frame of point cloud; is the estimated value of the i -th point in the current frame of point cloud; is the number of point features; is the position of the i -th point feature in the previous frame of point cloud; is the estimated value of the position of the i -th point feature in the current frame; is the number of line features; is the position of the i -th line feature in the previous frame of point cloud; is the estimated value of the position of the i -th line feature in the current frame.

[0057] Step C: Iteratively optimize the pose so that the value of the first loss function meets the requirements, and obtain the rotation R 1 and translation T 1 of the final reliable pose, R 1 = R · R 0, T 1 = T 0 - T ; where, R is the estimated rotation, T is the estimated translation, R 0 and T 0 are the rotation and translation values in the initial pose of the previous frame respectively.

[0058] In Embodiment 2, as Figure 5 shown, the pan-tilt linkage is as follows: when the boom of the ship unloader rotates, the two pan-tilts need to be synchronized so that the long-distance cameras can always look at the bow and stern of the ship. Each pan-tilt needs to carry three devices: a short-distance lidar, a 180° wide-angle camera, and a 70° long-distance camera. The specific process is as follows: Step a: When the boom of the ship unloader rotates, the two pan-tilts rotate accordingly, keeping the long-distance cameras always pointing in the directions of the bow and stern; when the pan-tilts rotate to a new angle, calibrate the pan-tilt angles.

[0059] Step b: Establish the point feature matching relationship ( , ) and the line feature matching relationship ( , ) between the current RGB image and the previous frame of RGB image.

[0060] Among them, is the point feature of the previous frame of RGB image, is the point feature of the current RGB image, is the line feature of the previous frame of RGB image, is the line feature of the current RGB image Step c: Construct a second loss function, which includes a second point cloud alignment loss function, a second point reprojection loss function, and a second line reprojection loss function.

[0061] In this embodiment, the first point cloud alignment loss function The calculation formula of

[0062] The second point reprojection loss function The calculation formula of is:

[0063] The second line reprojection loss function The calculation formula of is:

[0064] The second loss function The calculation formula of is:

[0065] In the formula, is the number of points in the point cloud; is the i th point in the previous frame of point cloud; is the estimated value of the i th point in the current frame of point cloud; is the number of point features; is the position of the i th point feature in the previous frame of point cloud; is the estimated value of the position of the i th point feature in the current frame; is the number of line features; is the position of the i th line feature in the previous frame of point cloud; is the estimated value of the position of the i th line feature in the current frame.

[0066] Step d: Iteratively optimize the pose so that the value of the second loss function meets the requirements, thereby calculating the final rotation angle of the pan-tilt R .

[0067] Although previous studies have proposed various technical solutions to address the problems of ship attitude measurement and material monitoring, in terms of the overall system cost and application efficiency, these methods generally have significant limitations such as poor economy and insufficient cost control.

[0068] Among them, the 3D reconstruction methods driven by mechanisms for 2D lidar scanning, although having acceptable measurement accuracy, rely on mechanical structures for multi-angle scanning, resulting in a slow scanning speed and being difficult to meet the real-time requirements of the high-frequency dynamic operation of ship loaders. Once the ship or material state changes, the system response lags, and the measurement data cannot be updated in a timely manner. As a result, operators cannot adjust the loading strategy based on real-time information, which not only reduces the operation efficiency but also increases the risk of misoperation, ultimately having a negative impact on the port operation efficiency and economic benefits. In addition, the mechanical drive system itself also significantly increases the equipment procurement, maintenance, and operation costs, with a relatively low overall cost-effectiveness.

[0069] Some other solutions choose to horizontally install a long-range lidar to reduce the system complexity and cost. However, such methods are restricted by the structure of the ship loader such as the chute and support, and there are large areas of blanks in the collected point cloud data, making it difficult to meet the accurate measurement requirements of the three-dimensional pose of the entire hull. Although some studies have attempted to horizontally install two lidars to compensate for the blind area caused by occlusion, in essence, they still cannot achieve a comprehensive observation of key areas such as the bow and stern. Their functions are limited to local measurements in the material area, and their ability to control the overall hull attitude is limited, and the investment is not proportional to the measurement effect.

[0070] Theoretically, there is also a concept of longitudinally installing two lidars to achieve measurement coverage directly above the hull. However, in practice, due to the relatively large distance between the front and rear of the hull and the lidar, the scanning resolution is severely insufficient, the measurement error is large, and it is difficult to achieve the high-precision measurement goal. At the same time, the required high-performance lidars and complex installation structures further increase the overall system cost, which is not conducive to large-scale popularization and application.

[0071] Some studies, out of cost-saving considerations, use low-cost, short-range lidars. However, such devices are easily interfered with in the bulk cargo loading environment with dense dust and complex reflections, have poor stability, and limited range, and cannot cover the distal areas of the hull such as the bow and stern, resulting in serious lack of measurement information. Although the cost is low, the practicality is extremely low.

[0072] There are also solutions that attempt to achieve a higher measurement speed by deploying multiple long-range lidars to meet the near-real-time dynamic operation requirements. However, their observation perspectives usually come from obliquely above, and it is difficult to obtain the complete point cloud information of the material pile at the open hatch. In addition, such systems not only have a large number of devices, complex integration, and heavy installation and maintenance workloads, but their high hardware investment and deployment costs also become a major obstacle to popularization and application. The overall economy is far inferior to the solution proposed in this embodiment.

[0073] In summary, traditional methods generally have deficiencies in cost control. They often sacrifice economy to improve performance and it is difficult to achieve a balance between performance and cost. On the basis of ensuring measurement accuracy and real-time performance, the solution of this embodiment significantly reduces the number of devices and system costs through a more concise and efficient system design, and has higher practical value and promotion prospects.

[0074] Therefore, the differences between this embodiment and traditional methods lie in the following four aspects: (1) Low-cost multi-view perception architecture, replacing the traditional high-cost fusion scheme of multiple long-distance lidars: Abandon the traditional high-cost long-distance lidar (such as 128-line lidar) scheme, and innovatively use a combination of low-cost cameras to replace it, significantly reducing the system hardware cost; Configure two low-cost short-distance lidars (with a 360° viewing angle) to cover the occlusion area around the chute, and combine a 180° wide-angle camera and a long-distance telephoto camera (which can cover a distance of 50 meters at the bow / stern) to build a multi-view data acquisition system; effectively solve the problems of occlusion blind spots and insufficient long-distance resolution existing in traditional single lidars, and achieve a more comprehensive and accurate spatial perception ability.

[0075] (2) Joint optimization algorithm based on image and point cloud fusion to achieve high-precision pose estimation: At the data processing level, this embodiment designs a joint loss function optimization mechanism for point cloud and image to achieve multi-modal data fusion and accuracy enhancement; fuse the lidar point cloud (M1) and the camera reconstruction model (M2) to construct a weighted loss function including a point cloud alignment term and an image feature back-projection alignment term.

[0076] (3) Pan-tilt linkage and dynamic micro-motion mechanism to achieve high-frequency pose update: Aiming at the perception delay problem caused by the frequent rotation of the boom during the operation of the ship unloader, this embodiment introduces high-precision pan-tilt linkage and dynamic micro-motion technology; during the rotation of the boom, the sensor adjusts through micro-motion to keep the long-distance camera always pointing in the direction of the bow and stern, ensuring real-time coverage of the key monitoring area and avoiding monitoring blind spots; the system can achieve real-time update of the pose. Compared with the multi-radar scanning delay (≥2 seconds) existing in traditional methods, this embodiment is more suitable for high-dynamic continuous operation scenarios.

[0077] (4) Multi-modal redundancy and anti-interference mechanism to adapt to complex port environments: To improve the stability and robustness of the system in a strong interference environment, this embodiment introduces multi-sensor redundancy design and various anti-interference mechanisms: The lidar uses an anti-dust coating and a dynamic filtering algorithm to improve the adaptability to port dust interference. Under typical interference factors such as material dispersion and glare, the system can still achieve stable and reliable data acquisition and pose calculation.

[0078] The effects after implementing the measurement system and method provided in this embodiment are as follows: In practical applications, the system operates efficiently and collaboratively according to the established process. Two short-range lidars perform full-coverage scanning from directly above, avoiding the occlusion problems caused by structures such as chutes and brackets in traditional lateral arrangements, ensuring that there are no visual dead angles or data holes in the hull model. The point cloud data is fused through motion compensation and spatial stitching algorithms to generate a complete and highly accurate three-dimensional model M1. At the same time, four cameras collect RGB images from different angles, and use point-line feature matching and SFM algorithms to reconstruct a three-dimensional model M2, and then obtain a hull model with higher fusion accuracy after model alignment.

[0079] On the basis of maintaining high measurement accuracy, this method fully realizes the acquisition of dynamic data and timing synchronization, and has good real-time performance and robustness. During the ship loading process, the system can instantly refresh the three-dimensional model according to the operation needs, accurately reflect the changes in the hull attitude and material volume, and provide a reliable basis for ship loading path planning and loading volume control.

[0080] Compared with traditional solutions that require multiple high-cost long-range lidars for fusion or use complex drive mechanisms, this method only uses two low-cost short-range lidars and two groups of low-cost camera modules, which can significantly reduce the overall cost of the system while ensuring accuracy, efficiency and real-time performance, and has strong engineering feasibility and application prospects.

[0081] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A ship unloader measurement system integrating lidar and camera, characterized in that, It includes two sets of vision and ranging components, which are symmetrically arranged on both sides of the chute of the ship unloader. Each set of vision and ranging components includes a wide-angle camera, a long-distance camera and a short-distance lidar; Among them, the wide-angle camera is used for 180° perspective coverage acquisition; the long-distance camera is used to improve the resolution of the bow and stern; the short-distance lidar is used to obtain high-density point cloud data at close range to make up for the information loss in the occluded area around the chute of the ship unloader.

2. The ship unloader measurement system integrating lidar and camera according to claim 1, characterized in that, The vision and ranging components are fixed on the pan-tilt of the ship unloader. The pan-tilt has an angle adjustment function with at least one degree of freedom, and the rotational angular velocity is controlled within the range of 0.05° / s to 2° / s, and the direction of its rotation axis is consistent with the longitudinal direction of the hull.

3. The ship unloader measurement system integrating lidar and camera according to claim 1, characterized in that, The wide-angle camera uses a wide-angle camera with a 180° downward perspective, the long-distance camera uses a long-distance camera with a 70° lateral perspective, and the short-distance lidar uses a short-distance lidar with a 360° omnidirectional perspective.

4. A measuring method for a ship unloader measuring system integrating lidar and camera as described in any one of claims 1 to 3, characterized in that, It includes the following steps: S10, Move the chute to directly above the empty ship, and then turn on the two short-distance lidars, two wide-angle cameras and two long-distance cameras installed on the chute seat. The pan-tilt is linked to keep the long-distance camera facing the bow or stern for the first attitude measurement of the hull; S20, Switch to the automatic operation mode. The ship unloader control system automatically controls the chute to move to a position close to the bow, and then move towards the stern at a constant speed; during the movement of the chute, whenever the position of the chute changes, the system will perform pan-tilt linkage and attitude measurement to ensure obtaining the complete information of the hull; after the scanning is completed, splice and integrate the scanning results obtained by the short-distance lidar to obtain the three-dimensional point cloud model M1; S30, Use a total of four cameras in two groups to collect the RGB images of the hull, use point-line features for matching, and use the SFM algorithm to obtain the three-dimensional reconstructed hull point cloud model M2; S40, Align M1 and M2; S50, During the actual process of loading, once there is a need to update the real-time state of the hull, obtain the updated hull model.

5. The measuring method according to claim 4, characterized in that, The attitude measurement is specifically: Establish the point feature matching relationship ( , ) and the line feature matching relationship ( , ) between the current RGB image and the previous frame RGB image; among them, is the point feature of the previous frame RGB image, is the point feature of the current RGB image, is the line feature of the previous frame RGB image, is the line feature of the current RGB image; Construct a first loss function, and the first loss function includes a first point cloud alignment loss function, a first point reprojection loss function and a first line reprojection loss function; Iteratively optimize the pose so that the value of the first loss function meets the requirements, and obtain the rotation in the final reliable pose R 1 and translation T 1, R 1 = R · R 0, T 1 = T 0 - T ; where R is the estimated rotation, T is the estimated translation, R 0 and T 0 are the rotation and translation values in the initial pose of the previous frame respectively.

6. The measuring method according to claim 5, characterized in that, The first point cloud alignment loss function has the following calculation formula: The first reprojection loss function has the following calculation formula: The first line reprojection loss function has the following calculation formula: The first loss function has the following calculation formula: Wherein, is the number of points in the point cloud; is the i -th point in the previous frame of the point cloud; is the estimated value of the i -th point in the current frame of the point cloud; is the number of point features; is the position of the i -th point feature in the previous frame of the point cloud; is the estimated value of the position of the i -th point feature in the current frame; is the number of line features; is the position of the i -th line feature in the previous frame of the point cloud; is the estimated value of the position of the i -th line feature in the current frame.

7. The measurement method according to claim 4, wherein The pan-tilt linkage is: when the boom of the ship unloader rotates, the two pan-tilts need to be synchronized and linked so that the long-distance camera always looks at the bow and stern.

8. The measurement method according to claim 7, characterized in that, The pan-tilt linkage is specifically: When the boom of the ship unloader rotates, the two pan-tilts rotate accordingly to keep the long-distance camera always pointing in the directions of the bow and stern; when the pan-tilt rotates to a new angle, calibrate the rotation angle of the pan-tilt; Establish the point feature matching relationship ( , ) and the line feature matching relationship ( , ) between the current RGB image and the previous frame RGB image; wherein, is the point feature of the previous frame RGB image, is the point feature of the current RGB image, is the line feature of the previous frame RGB image, is the line feature of the current RGB image; Construct a second loss function, and the second loss function includes a second point cloud alignment loss function, a second point reprojection loss function and a second line reprojection loss function; Iteratively optimize the pose so that the value of the second loss function meets the requirements, thereby calculating the rotation angle of the final pan-tilt head R .

9. The measurement method according to claim 8, wherein The second point cloud alignment loss function has the following calculation formula: The second reprojection loss function has the following calculation formula: The second line reprojection loss function has the following calculation formula: The second loss function has the following calculation formula: Wherein, is the number of points in the point cloud; is the i -th point in the previous frame of the point cloud; is the estimated value of the i -th point in the current frame of the point cloud; is the number of point features; is the position of the i -th point feature in the previous frame of the point cloud; is the estimated value of the position of the i -th point feature in the current frame; is the number of line features; is the position of the i -th line feature in the previous frame of the point cloud; is the estimated value of the position of the i -th line feature in the current frame.

Images