Excavator automatic digging method based on image recognition

Abstract

The application provides a kind of based on image recognition's automatic excavator digging method, based on material pool and excavator is realized, the upper edge of material pool is equipped with laser radar and binocular camera, and excavator contains excavator arm, double bucket assembly and control unit.The method is: laser radar sampling material pile three-dimensional point cloud determines wave peak initial position, camera secondary calibration obtains wave peak final position and slope angle;Control unit switches first / second digging mode according to wave peak and excavator distance, first mode controls bucket wall digging and detects bottom by pressure sensor, second mode adjusts bucket angle according to slope angle;There is also a corresponding digging scheme when there is no wave peak, and the point cloud is updated after each digging;The application improves the positioning accuracy and environmental adaptability of wave peak, optimizes the digging efficiency, reduces manual intervention, and adapts to grain, sand and other material digging.

Description

Technical Field

[0001] This application relates to the field of image recognition application technology, and in particular to an automatic excavation method for excavators based on image recognition. Background Technology

[0002] A material storage pit is a device used to store raw materials, typically grains, sand, and gravel. Because it is located below ground level, excavation operations largely rely on excavators. During storage, unloading is usually done by trucks, creating a material pile within the pit, which forms a wave-like peak. Currently, excavation is mostly done manually with excavators, but this relies on operator experience and is complex due to the pit's depth. To improve automation, some solutions attempt to use sensors to detect the material pile and automate excavation. However, existing excavators have a relatively simple excavation mode, making it difficult to adapt to different excavation strategies based on the varying distances between the material pile peaks and the excavator, resulting in low excavator efficiency. Summary of the Invention

[0003] The purpose of this invention is to solve the above-mentioned problems by providing an automatic excavation method for excavators based on image recognition.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides an automatic excavation method based on image recognition, the method being implemented based on a material pool and an excavator deployed next to the material pool;

[0006] The excavator includes: a digging arm and two buckets that can open and close to each other; a pressure sensor for detecting the pressure acting on the bucket lips; a camera assembly and a lidar mounted on the base of the excavator facing into the material pool; and a control unit for controlling the digging arm and buckets to dig up the material.

[0007] The mining method includes the following steps:

[0008] S1, the position of the wave crest of the material in the material pool and the straight-line distance L1 between the wave crest and the side of the material pool near the excavator are obtained by the camera assembly and lidar;

[0009] S2, determine whether the straight distance L1 is greater than or equal to the effective digging width D of the bucket. If yes, use the second digging mode to dig the material; otherwise, use the first digging mode to dig the material.

[0010] The first digging mode is as follows: the bucket is controlled to dig downwards close to the inner wall of the material pool on the side close to the excavator. During this process, the pressure sensor determines whether the bottom has been reached. After the bottom has been reached, the process returns to step S1.

[0011] The second digging mode is as follows: after adjusting the angle of the buckets so that the bucket lips are both facing the slope on the side of the material closer to the excavator, the two buckets are controlled to insert into the material for digging.

[0012] Based on the above structural improvements, in step S1, the camera assembly and the lidar also acquire the slope angle of the material in the area where the wave crest is located.

[0013] The excavator also includes an angle sensor for detecting the angle of entry of the bucket.

[0014] The second digging mode also includes adjusting the angle of the buckets according to the slope angle, calibrating the angle sensor so that the bucket lips are all facing the slope of the material closer to the excavator, and then controlling the two buckets to insert into the material for digging.

[0015] Based on the above structural improvements, the camera assembly includes a binocular camera;

[0016] The camera assembly and lidar acquire the position of the wave crest of the material in the pool, including:

[0017] The LiDAR scanner scans the material pile within the material pool, generating two-dimensional coordinates (x, y) of all points on the pile surface relative to the excavator. Point cloud preprocessing filters out background and noise points outside the material pool, retaining valid points on the pile surface. Statistical filtering removes isolated points. The material pool plane is divided into 1m×1m grids, and valid points are assigned to their corresponding grids. The average height H(i, j) of each grid and the overall average height Havg of the material pile are calculated. Grids with H(i, j) > Havg are selected from the initial peak regions and clustered into continuous regions. The center coordinates (x1, y1) of each region are taken as the initial peak position, where i represents the row number of the grid in the material pool plane and j represents the column number of the grid.

[0018] The method for obtaining the position of the wave crest of the material in the pool using the camera assembly and lidar further includes: performing secondary calibration on the initial position of the wave crest using the camera assembly to obtain the final position of the wave crest.

[0019] The camera component, based on the initial position (x1, y1) output by the LiDAR, verifies and corrects the coordinates using visual features, and outputs the final peak position, including:

[0020] The camera and LiDAR are synchronized via a time sync device to ensure consistent data acquisition timing. Based on the camera's preset calibration parameters, the initial peak coordinates (x1, y1) of the LiDAR are mapped to the two-dimensional image pixel coordinates (u, v) of the camera, locating the corresponding peak candidate region in the image. A 50×50 pixel sub-region surrounding (u, v) in the image is extracted, and the visual characteristics of this region are analyzed.

[0021] By using the parallax of the binocular cameras, the relative height gradient of the area is calculated to confirm whether the center position (u,v) is the highest point in the area; if the visual feature verification shows that (x1,y1) is the true highest point of the peak, the coordinates are directly retained.

[0022] If the visual system detects a shift in the highest point within the region, the shifted two-dimensional coordinates (x2, y2) are deduced through coordinate mapping; the output (x2, y2) is then used as the final position of the peak.

[0023] Using the excavator's coordinates as the origin (0,0), calculate the straight-line distance L1 based on the final position coordinates (x2,y2) of the wave crest.

[0024] Methods for obtaining the slope angle of a wave crest from its final position include:

[0025] Centered on the final position of the wave peak, a rectangular calculation area is formed by extending towards the excavator. The length of the rectangular calculation area is set along the slope extension direction, and the width is set perpendicular to the slope extension direction. From the 3D point cloud of the material pile acquired by the lidar, point clouds whose coordinates fall within the rectangular calculation area are selected to form a local point cloud subset. The local point cloud subset is denoised to remove isolated points and retain effective point clouds that can reflect the surface state of the material pile slope. The denoised effective point clouds are subjected to plane fitting to obtain a fitting plane that matches the surface morphology of the material pile slope. The angle between the fitting plane and the horizontal plane is calculated, and this angle is the slope angle of the corresponding slope area of ​​the material pile.

[0026] During the excavation process, the camera component transmits excavation images in real time for remote monitoring by humans.

[0027] Based on the above structural improvements, the two buckets are respectively a first bucket and a second bucket, and the first bucket and the second bucket form a bucket assembly, wherein the back of the first bucket is arranged in a straight line structure, and the back of the second bucket is arranged in an arc shape structure.

[0028] Based on the above structural improvements, the bucket assembly further includes:

[0029] A first upper arm and a second upper arm, wherein the first upper arm is mounted on the first bucket and the second upper arm is mounted on the second bucket, and the first upper arm and the second upper arm are hinged to each other;

[0030] The support frame has a hinge point where the first upper arm and the second upper arm are hinged to each other, which is rotatably connected to the inner side of the support frame via a rotating shaft.

[0031] A first hydraulic cylinder and a second hydraulic cylinder, wherein the cylinder end of the first hydraulic cylinder is hinged to the inner top side of the support frame, and the telescopic end is hinged to the upper end of the first upper arm; the cylinder end of the second hydraulic cylinder is hinged to the inner top side of the support frame, and the telescopic end is hinged to the upper end of the second upper arm; and two pressure sensors are provided, which are respectively installed at the pressure measuring ports of the first hydraulic cylinder and the second hydraulic cylinder.

[0032] A connecting frame, the front end of which is rotatably connected to the support frame, and the rear end of which is rotatably connected to the excavator's boom;

[0033] The third hydraulic cylinder has its cylinder end hinged to the inner side of the top of the connecting frame, and its telescopic end hinged to the outer side of the support frame.

[0034] The control unit is connected to and controls the excavator arm, the first hydraulic cylinder, the second hydraulic cylinder, and the third hydraulic cylinder. The control unit drives the opening and closing of the first bucket and the second bucket by controlling the extension and retraction of the first hydraulic cylinder and / or the second hydraulic cylinder. The control unit adjusts the orientation of the first bucket and the second bucket by controlling the extension and retraction of the third hydraulic cylinder.

[0035] Based on the above structural improvements, the first mining mode includes the following steps:

[0036] S11. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to open the first bucket and the second bucket to the maximum angle, adjusts the action of the digging arm so that the back of the first bucket is in close contact with the inner wall of the material pool, and drives the entire bucket assembly to move vertically downward so that the first bucket and the second bucket are inserted into the material pile.

[0037] S12. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to close the first bucket and the second bucket together, and then controls the excavator's arm to pull out the bucket assembly to complete the digging.

[0038] In step S11, when the bucket assembly moves downward, if it is determined that the bucket has touched the bottom, the downward movement of the bucket is stopped. Then, in step S12, during the process of the first bucket and the second bucket closing with each other, the bucket lips of the first bucket and the second bucket are ensured to be in contact with the inner bottom surface of the material pool.

[0039] Based on the improvements to the above structure, the second mining mode includes the following steps:

[0040] S21. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to open the first bucket and the second bucket to the maximum angle, and adjusts the angle of the bucket assembly by the third hydraulic cylinder so that the bucket lips of the first bucket and the second bucket face the slope closer to the excavator. The control unit controls the arm so that the bucket lips of the first bucket and the second bucket are inserted into the slope facing the slope closer to the excavator.

[0041] S22. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to close the first bucket and the second bucket together, and then controls the excavator's arm to pull out the bucket assembly to complete the digging.

[0042] Based on the above structural improvements, the lidar defines the wave peak as a region with a relative height ≥ 0.5m or a slope angle > 30°.

[0043] The relative height is the difference between the overall average height Havg of the stockpile and the height of the target stockpile.

[0044] The excavation method further includes: if no wave peak is detected, for areas outside the effective excavation width D of the bucket, the following steps are used for excavation:

[0045] S311. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to open the first bucket and the second bucket to the maximum angle. The angle of the bucket assembly is adjusted by the extension and retraction of the third hydraulic cylinder so that the bucket lips of the first bucket and the second bucket face the surface of the material pile. The control unit controls the digging arm so that the bucket lips of the first bucket and the second bucket are inserted into the material pile at a vertical angle.

[0046] S312. The control unit controls the first bucket and the second bucket to close together, and then controls the excavator's arm to pull out the bucket assembly to complete the digging.

[0047] For areas within the effective digging width D of the bucket, the first digging mode is used for digging.

[0048] The advantages or beneficial effects of the above technical solutions include at least the following:

[0049] This invention utilizes the collaborative operation of a lidar and a binocular camera assembly. The lidar acquires 3D point cloud data of the material pile to initially locate the peak, while the binocular camera performs secondary calibration to correct the peak position and assist in verifying the slope angle. This effectively improves the accuracy of peak positioning and the reliability of slope angle calculation. Simultaneously, the camera serves as a redundant sensing tool. When the lidar data is abnormal due to strong light or high dust levels, visual images assist in confirming the material pile outline and slope direction, preventing system downtime due to single sensor failure and significantly improving operational adaptability in complex environments. Furthermore, by setting a dual excavation mode based on the distance between the peak and the excavator, the first excavation mode is adapted to peaks farther away. For wall-hugging digging scenarios where the effective digging width D of the bucket is outside the effective digging width D of the bucket, the second digging mode is adapted to slope digging scenarios where the wave crest is outside the effective digging width D of the bucket. It also combines a pressure sensor to realize bucket bottom contact detection and an angle sensor to calibrate the bucket insertion angle, which can be specifically adapted to the digging needs of different material pile positions, reducing missed digging, empty digging and material disturbance. The bucket assembly has improved the bucket rotation torque and opening and closing angle compared with the existing technology through a special hinge structure between the upper arm and the hydraulic cylinder. Under the same hydraulic cylinder thrust, it can achieve more stable bucket opening and closing and material grabbing. At the same time, the cooperation between the connecting frame, motor and gears makes it easy to adjust the bucket wall-hugging angle, optimizing the digging execution effect.

[0050] Furthermore, this invention re-collects point cloud data and updates the peak position after each excavation, ensuring real-time accuracy of the excavation data. It also sets up corresponding excavation schemes for scenarios without peaks. Combined with the remote monitoring function of the camera, it improves the automation level, operation efficiency and reliability of material excavation in the material pool, reduces the cost of manual intervention, and reduces the risk of material damage and excavator component wear. Attached Figure Description

[0051] The accompanying drawings illustrate exemplary embodiments of the present application and, together with the description thereof, serve to explain the principles of the present application. These drawings are included to provide a further understanding of the present application and are incorporated in and constitute a part of this specification.

[0052] Figure 1 A schematic diagram showing the positional relationship between the excavator and the material pool according to an embodiment of the present invention is shown;

[0053] Figure 2 A schematic diagram of the structure of the bucket assembly according to an embodiment of the present invention is shown;

[0054] Figure 3 A schematic diagram showing the change in bucket orientation after the third hydraulic cylinder extends and retracts according to an embodiment of the present invention is shown;

[0055] Figure 4 A schematic diagram of the assembly of the bucket assembly according to an embodiment of the present invention is shown;

[0056] Figure 5An assembly diagram of the first and second buckets according to an embodiment of the present invention is shown;

[0057] Figure 6 The figure shows a schematic diagram of the steps of the first digging mode of the present invention. In the figure, step a is to press the back of the first bucket against the side wall of the material pool, step b is to scoop the first bucket and the second bucket into the material and the bucket lip touches the bottom, and steps c and d are to close the first bucket and the second bucket to complete the material digging.

[0058] Figure 7 The diagram illustrates a second digging mode according to an embodiment of the present invention, in which the bucket is oriented toward a slope, and the dashed arrows in the diagram indicate the digging direction of the bucket.

[0059] Figure 8 A schematic diagram showing that the wave crest appears outside the excavation width D according to an embodiment of the present invention is illustrated;

[0060] Figure 9 This diagram illustrates a scenario where the wave crest appears within the excavation width D according to an embodiment of the present invention.

[0061] Figure 10 The diagram shows a rotating platform of an excavator according to an embodiment of the present invention rotating toward a target area and then rotating a connecting frame such that the first bucket is in close contact with the side wall of the hopper.

[0062] Figure 11 The diagram shows a comparison of the opening angle of the bucket assembly of this application and the bucket assembly of the prior art, with the upper part of the figure showing the bucket assembly of this application and the lower part showing the bucket assembly of the prior art.

[0063] Figure 12 The diagram shows a comparison of the digging directions of the first digging mode and the second digging mode under an embodiment of the present invention. The upper part of the diagram represents the first digging mode, and the lower part represents the second digging mode. The blue arrows indicate the digging direction of the bucket, and the red arrows indicate the dumping direction of the material pile after digging.

[0064] Reference numerals: 10, material pool; 20, excavator; 21, excavator arm; 22, bucket assembly; 221, first bucket; 222, second bucket; 223, support frame; 224, connecting frame; 225, first hydraulic cylinder; 226, second hydraulic cylinder; 227, third hydraulic cylinder; 2201, first upper arm; 2202, second upper arm; 23, base. Detailed Implementation

[0065] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0066] It should be noted that, where there is no conflict, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0067] It should be understood that the term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0068] It should be noted that the terms "one" and "more" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0069] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0070] An embodiment of the present invention provides an automatic excavation method based on image recognition, the method being implemented based on a material pool 10 and an excavator 20 deployed next to the material pool 10;

[0071] like Figure 1 The excavator 20 shown includes: a digging arm 21 and two buckets that can open and close to each other; a pressure sensor for detecting the pressure acting on the bucket lips; a camera assembly and a lidar facing into the material pool 10 are mounted on the base 23 of the excavator 20; and a control unit for controlling the digging arm 21 and buckets to dig up materials; the digging method includes the following steps:

[0072] The mining method includes the following steps:

[0073] S1, the position of the wave crest of the material in the material pool 10 and the straight-line distance L1 between the wave crest and the side of the material pool 10 near the excavator 20 are obtained by the camera assembly and lidar.

[0074] S2, determine whether the straight distance L1 is greater than or equal to the effective digging width D of the bucket. If yes, the second digging mode is used to dig the material; otherwise, the first digging mode is used to dig the material. It should be noted that the effective digging width D of the bucket is the digging width after the two buckets are opened to the maximum angle and the material is dug and closed with one shovel, not the maximum working range after the digging arm 21 is extended.

[0075] The first mining mode is as follows: Figure 12 As shown in the upper part, the bucket is controlled to dig downwards close to the inner wall of the material pool 10 near the excavator 20. During this process, the pressure sensor determines whether it has reached the bottom. After reaching the bottom, the process returns to step S1. In the figure, the blue arrow indicates the digging direction, and the red arrow indicates the direction in which the material at the crest of the wave collapses after digging because it loses the support of the material in the blue arrow area.

[0076] The second mining mode is: such as Figure 12 As shown in the lower half, the angle of the buckets is adjusted so that the bucket lips face the slope on the side of the material closer to the excavator 20, and then the two buckets are controlled to insert into the material to dig. In the figure, the blue arrows indicate the digging direction, and the red arrows indicate the direction in which the material at the crest of the wave collapses after digging because it loses the support of the material in the blue arrow area.

[0077] Whether it is the first digging mode or the second digging mode, the goal is to dig the part of the material that is as close to the excavator 20 as possible, so as to minimize the extension distance of the excavator arm 21 and shorten the digging time of a single digging.

[0078] The camera assembly includes a binocular camera;

[0079] The process involves using lidar to collect three-dimensional point cloud data of the material pile within the material pool 10 to determine the initial position of the material's peak. Then, a camera assembly is used to perform secondary calibration on the initial position of the peak. In environments with strong light (such as direct sunlight on the material pile at noon, causing reflection interference) or high dust (dust during sand and gravel excavation), the point cloud quality of the lidar may decrease (fewer effective points, more noise). In such cases, the camera assembly can serve as an auxiliary sensing method to reduce the system's misjudgment rate.

[0080] The method for obtaining the final position of the wave crest described above includes: wherein the camera assembly and lidar acquire the position of the wave crest of the material in the pool by:

[0081] The LiDAR scanner scans the material pile within the material pool, generating two-dimensional coordinates (x, y) of all points on the pile surface relative to the excavator. Point cloud preprocessing filters out background and noise points outside the material pool, retaining only valid points on the pile surface. Statistical filtering removes isolated points. The material pool plane is divided into 1m×1m grids, assigning valid points to their corresponding grids. The average height H(i, j) of each grid and the overall average height Havg of the material pile are calculated. Initial peak regions are selected from grids where H(i, j) > Havg, clustered into continuous regions, and the center coordinates (x1, y1) of each region are taken as the initial peak position. Here, i represents the row number of the grid in the material pool plane, and j represents the column number. Specifically, i corresponds to the grid number in the y-axis direction (row), j corresponds to the grid number in the x-axis direction (column), and H(i, j) is the average height of all valid point cloud points within the grid at row i and column j, used for subsequent comparison with the overall average height of the material pile to select peak regions.

[0082] The method for acquiring the position of the wave crest of the material in the pool using the camera assembly and lidar further includes: performing secondary calibration on the initial position of the wave crest using the camera assembly to obtain the final position of the wave crest.

[0083] The camera component, based on the initial position (x1, y1) output by the LiDAR, verifies and corrects the coordinates using visual features, and outputs the final peak position, including:

[0084] The camera and LiDAR are synchronized via a time sync device to ensure consistent data acquisition timing. Based on the camera's preset calibration parameters, the initial peak coordinates (x1, y1) of the LiDAR are mapped to the two-dimensional image pixel coordinates (u, v) of the camera, locating the corresponding peak candidate region in the image. A 50×50 pixel sub-region surrounding (u, v) in the image is extracted, and the visual characteristics of this region are analyzed.

[0085] By using the parallax of the binocular cameras, the relative height gradient of the area is calculated to confirm whether the center position (u,v) is the highest point in the area; if the visual feature verification shows that (x1,y1) is the true highest point of the peak, the coordinates are directly retained.

[0086] If the visual system detects a shift in the highest point within the region, the shifted two-dimensional coordinates (x2, y2) are deduced through coordinate mapping; the output (x2, y2) is then used as the final position of the peak.

[0087] Using the excavator's coordinates as the origin (0,0), calculate the straight-line distance L1 based on the final position coordinates (x2,y2) of the wave crest.

[0088] Methods for obtaining the slope angle of a wave crest from its final position include:

[0089] Centered on the final position of the wave peak, a rectangular calculation area is formed by extending towards the excavator. The length of the rectangular calculation area is set along the slope extension direction, and the width is set perpendicular to the slope extension direction. From the 3D point cloud of the material pile acquired by the lidar, point clouds whose coordinates fall within the rectangular calculation area are selected to form a local point cloud subset. The local point cloud subset is denoised to remove isolated points and retain effective point clouds that can reflect the surface state of the material pile slope. The denoised effective point clouds are subjected to plane fitting to obtain a fitting plane that matches the surface morphology of the material pile slope. The angle between the fitting plane and the horizontal plane is calculated, and this angle is the slope angle of the corresponding slope area of ​​the material pile.

[0090] The method for obtaining the slope angle of a wave crest through its final position further includes: adding at least two adjacent rectangular verification areas around the final position of the wave crest, repeating the above steps, and calculating the slope angle corresponding to each verification area respectively; performing a weighted average processing on the slope angle and the slope angle corresponding to each verification area, with the weighting weights distributed according to the distance between the area and the final position of the wave crest, the closer the distance, the higher the weight, to obtain the final material pile slope angle;

[0091] Specifically, let the main calculation area centered on the final position of the wave crest be A0, and the slope angle calculated therein be θ0; the two additional adjacent verification areas are A1 and A2, and the corresponding slope angles calculated are θ1 and θ2. If more verification areas are added later, they can be expanded according to this logic, such as A3, θ3, etc.

[0092] I. Setting distance parameters:

[0093] Let d0, d1, and d2 be the straight-line distances from the center point of each region to the final position of the wave crest. Key rule: The center point of the main region A0 is the final position of the wave crest, so d0 = 0 (closest distance). A1 and A2 are adjacent regions, and the distance satisfies d1 < d2 (or can be adjusted according to the actual layout, just ensure that "the closer to the wave crest, the smaller the d value").

[0094] II. Mathematical Model Calculation

[0095] Step 1: Calculate the weights of each region (W0, W1, W2)

[0096] The core logic of weighting is: the smaller the distance d, the larger the weight W, and the sum of the weights of all regions is 1.

[0097] The basic weights are calculated using the "reciprocal of distance," and then "normalization" is applied to ensure that the sum of the weights is 1, as detailed below:

[0098] 1. First, calculate the basic weights:

[0099] Main region A0: Since d0=0, its basic weight is directly set to a fixed value (such as K, it is recommended to take 2 to ensure that its weight is maximized).

[0100] Validation region A1: Basic weight = 1 / d1;

[0101] Validation region A2: Basic weight = 1 / d2;

[0102] 2. Then perform weight normalization (ensuring W0 + W1 + W2 = 1):

[0103] Total base weight = K + (1 / d1) + (1 / d2);

[0104] Main region weight: W0 = K / [K + (1 / d1) + (1 / d2)];

[0105] Validation region A1 weight: W1=(1 / d1) / [K+(1 / d1)+(1 / d2)];

[0106] Weight of verification region A2: W2=(1 / d2) / [K+(1 / d1)+(1 / d2)];

[0107] Step 2: Calculate the final slope angle (θ_final)

[0108] The final result is obtained by weighted summation of "weight × slope angle":

[0109] θ_final=W0×θ0+W1×θ1+W2×θ2;

[0110] Example illustration: Assume a real-world scenario where:

[0111] Main region A0: θ0=35°, d0=0, take K=2;

[0112] Verification area A1: distance d1=1m, θ1=33°;

[0113] Verification area A2: distance d2=2m, θ2=31°;

[0114] Step 1: Calculate the weights:

[0115] 1. Basic weights:

[0116] A0 base weight = 2; A1 base weight = 1 / 1 = 1; A2 base weight = 1 / 2 = 0.5;

[0117] Total base weight = 2 + 1 + 0.5 = 3.5;

[0118] 2. Normalized weights:

[0119] W0 = 2 / 3.5 ≈ 0.57 (57%); W1 = 1 / 3.5 ≈ 0.29 (29%); W2 = 0.5 / 3.5 ≈ 0.14 (14%); This conforms to the principle that "the closer the distance, the higher the weight": A0 > A1 > A2;

[0120] Step 2: Calculate the final slope angle:

[0121] θ_final=0.57×35°+0.29×33°+0.14×31°≈33.8°.

[0122] It is important to note that the quality of point cloud acquisition by lidar is easily degraded in environments with strong light (such as direct sunlight on a material pile at noon, which can cause reflected signals to interfere with the quality of point clouds) and high dust (such as dust generated during sand and gravel excavation that can obscure the laser), which may lead to failure in peak location and slope angle calculation. The camera component can serve as a redundant sensing means. When lidar data is abnormal, it can rely on visual images (which can still identify the general outline of the material pile even in strong light / dust environments) to help confirm the approximate location of the peak and determine the slope direction, thus avoiding the shutdown of the entire system due to the failure of a single sensor and improving the adaptability of the solution in complex operating environments.

[0123] During the excavation process, the camera component can also transmit excavation images in real time for remote monitoring by humans.

[0124] The first digging mode described above involves controlling one of the buckets to dig downwards close to the inner wall of the material pool 10 near the excavator 20. During the digging process in the first digging mode, the value of the pressure sensor is detected in real time. When the preset value is reached, it is determined that the bucket lip has touched the bottom and the bucket stops moving downwards.

[0125] The second digging mode is to adjust the angle of the buckets according to the slope angle, and calibrate it through the angle sensor so that the bucket lips are facing the slope of the material closer to the excavator 20, and then control the two buckets to insert into the material for digging.

[0126] When lidar calculates the slope angle by fitting a local point cloud plane, if there are local collapses (such as depressions formed by loose grain accumulation) or protrusions (such as local agglomerations of sand and gravel) on the surface of the material pile, the fitted plane may deviate from the actual slope shape. The camera can extract the visual contour line corresponding to the slope calculation area and use the angle between the contour line and the horizontal line to help verify the slope angle calculated by lidar. If the deviation between the two exceeds the threshold, it can trigger lidar to refit the point cloud to ensure that the final slope angle is consistent with the actual slope shape of the material pile. This provides a reliable basis for "adjusting the bucket angle according to the slope angle" in the second digging mode and avoids insufficient bucket insertion depth or excessive disturbance of the material pile due to slope angle deviation.

[0127] After each excavation is completed, the three-dimensional point cloud data of the material pile is re-acquired by lidar to update the position of the peak, instead of excavating based on the data collected the first time, until the material removal operation is completed.

[0128] like Figure 2 As shown, the two buckets mentioned above are the first bucket 221 and the second bucket 222, which together form the bucket assembly 22. The back of the first bucket 221 is arranged in a straight line, while the back of the second bucket 222 is arranged in an arc shape.

[0129] Bucket assembly 22 also includes:

[0130] The first upper arm 2201 and the second upper arm 2202 are hinged to each other.

[0131] The hinge point where the first upper arm 2201 and the second upper arm 2202 are hinged to each other is rotatably connected to the inner side of the support frame 223 through a rotating shaft;

[0132] like Figure 4 , Figure 5 and Figure 7 As shown, there is a first hydraulic cylinder 225 and a second hydraulic cylinder 226. The cylinder end of the first hydraulic cylinder 225 is hinged to the inner top side of the support frame 223, and the telescopic end is hinged to the upper end of the first upper arm 2201. The cylinder end of the second hydraulic cylinder 226 is hinged to the inner top side of the support frame 223, and the telescopic end is hinged to the upper end of the second upper arm 2202. There are two pressure sensors, which are respectively installed at the pressure measuring ports of the first hydraulic cylinder 225 and the second hydraulic cylinder 226. The pressure from the bucket lip will be transmitted to the first hydraulic cylinder 225 and the second hydraulic cylinder 226. The pressure measuring port indicates whether the bucket lip has touched the bottom.

[0133] like Figure 2 As shown, a first hydraulic cylinder 225 is mounted above the second bucket 222, driving the first bucket 221 to rotate. A second hydraulic cylinder 226 is mounted above the first bucket 221, driving the second bucket 222 to rotate. The two hydraulic cylinders can be controlled independently. This design also increases the torque required to rotate the first and second buckets 221 and 222 via the first and second upper arms 2201 and 2202. Compared to existing technology that directly hinges the telescopic ends of the hydraulic cylinders to the bucket back, this design not only achieves greater torque under the same hydraulic cylinder thrust but also increases the bucket's opening and closing angle. Figure 11 As shown.

[0134] The connecting frame 224 is rotatably connected to the support frame 223 at its front end. An angle sensor is installed at the pivot of the connecting frame 224 and the support frame 223 to identify the bucket's entry angle. The rear end is rotatably connected to the excavator arm 21 of the excavator 20. Specifically, a motor is installed at the bottom of the connecting frame 224, and a pivot is fixed at the front end of the excavator arm 21. The pivot rotatably passes through the upper end of the connecting frame 224 and is equipped with a first gear. A second gear that meshes with the first gear is installed at the output end of the motor, thereby rotating the connecting frame 224 (the motor, the first gear, and the second gear are not shown in the figure).

[0135] In the first digging mode, one of the buckets needs to be pressed against the side wall of the material pool 10. Therefore, the rotating platform of the excavator 20 needs to rotate towards the target area on both sides, and then the output end of the drive motor rotates, driving the second gear through the first gear to rotate, thereby realizing the rotation of the connecting frame 224. This allows the bucket back of one of the buckets to be pressed against the side wall of the material pool 10. Figure 10 As shown.

[0136] The third hydraulic cylinder 227 has its cylinder end hinged to the inner side of the top of the connecting frame 224, and its telescopic end hinged to the outer side of the support frame 223.

[0137] The control unit is connected to and controls the excavator arm 21, the first hydraulic cylinder 225, the second hydraulic cylinder 226 and the third hydraulic cylinder 227. The control unit drives the opening and closing of the first bucket 221 and the second bucket 222 by controlling the extension and retraction of the first hydraulic cylinder 225 and / or the second hydraulic cylinder 226. The control unit adjusts the orientation of the first bucket 221 and the second bucket 222 by controlling the extension and retraction of the third hydraulic cylinder 227.

[0138] The first mining mode described above includes the following steps:

[0139] S11, the control unit controls the first hydraulic cylinder 225 and / or the second hydraulic cylinder 226 to open the first bucket 221 and the second bucket 222 to their maximum angle, adjusts the movement of the digging arm 21 so that the back of the first bucket 221 is pressed tightly against the inner wall of the material pool 10, and drives the entire bucket assembly 22 to move vertically downward, so that the first bucket 221 and the second bucket 222 are inserted into the material pile, such as... Figure 6 Steps a to b are shown below;

[0140] S12, the control unit controls the first hydraulic cylinder 225 and / or the second hydraulic cylinder 226 to close the first bucket 221 and the second bucket 222 together. After controlling the excavator arm 21 to pull out the bucket assembly 22, the digging is completed. Figure 6 Steps c through d are shown;

[0141] In step S11, when the bucket assembly 22 moves downward, if it is determined that the bucket has touched the bottom, the bucket stops moving downward. Then, in step S12, during the process of the first bucket 221 and the second bucket 222 closing with each other, the bucket lips of the first bucket 221 and the second bucket 222 are ensured to be in contact with the inner bottom surface of the material pool 10.

[0142] The second mining mode includes the following steps:

[0143] S21. The control unit controls the first hydraulic cylinder 225 and / or the second hydraulic cylinder 226 to open the first bucket 221 and the second bucket 222 to the maximum angle. The third hydraulic cylinder 227 adjusts the angle of the bucket assembly 22 so that the bucket lips of the first bucket 221 and the second bucket 222 face the slope closer to the excavator 20. The control unit controls the arm 21 so that the bucket lips of the first bucket 221 and the second bucket 222 are inserted into the slope facing the slope closer to the excavator 20.

[0144] S22, the control unit controls the first hydraulic cylinder 225 and / or the second hydraulic cylinder 226 to close the first bucket 221 and the second bucket 222 together, and then controls the excavator arm 21 to pull out the bucket assembly 22 to complete the digging.

[0145] like Figure 8 As shown, if the calculated position of the wave crest is at point A, and it is determined that point A1 is the slope of the wave crest near the excavator 20, and the distance from point A to the side of the hopper 10 near the excavator 20 is the actual straight-line distance L1, it is clearly within the effective digging width D of the bucket. The control unit controls the excavator 20 to dig at the position of point A1 in the second digging mode; Figure 9 As shown, if the calculated position of the wave crest is located at point B, and the distance from point B to the side of the material pool 10 near the excavator 20 is the actual straight-line distance L1, it is clear that it has not invaded the effective digging width D of the bucket, and point B is near the side of the material pool 10 near the excavator 20. The control unit controls the excavator 20 to dig at the position of point B in the first digging mode.

[0146] If the wave crest is outside the maximum working range of excavator 20, move excavator 20 until the wave crest is within the maximum working range.

[0147] Furthermore, the lidar defines the peak as the area with a relative height ≥ 0.5m or a slope angle > 30°.

[0148] The relative height is the difference between the overall average height of the stockpile, Havg, and the height of the target stockpile.

[0149] The excavation method further includes: if no wave peak is detected, for areas outside the effective excavation width D of the bucket, the following steps are performed for excavation:

[0150] S311, the control unit controls the first hydraulic cylinder 225 and / or the second hydraulic cylinder 226 to open the first bucket 221 and the second bucket 222 to their maximum angle, and adjusts the angle of the bucket assembly 22 by extending and retracting the third hydraulic cylinder 227, such as... Figure 3 As shown, the bucket lips of the first bucket 221 and the second bucket 222 face the surface of the material pile, and the control unit controls the digging arm 21 so that the bucket lips of the first bucket 221 and the second bucket 222 are inserted into the material pile at a vertical angle.

[0151] S312, The control unit controls the first bucket 221 and the second bucket 222 to close together, and after controlling the excavator arm 21 of the excavator 20 to pull out the bucket assembly 22, the digging is completed;

[0152] For areas within the effective digging width D of the bucket, the first digging mode is used for digging.

[0153] In the description of this application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0154] Those skilled in the art should understand that the above embodiments are merely for illustrative purposes and are not intended to limit the scope of this application. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of this application.

Claims

1. An automatic excavation method for excavators based on image recognition, characterized in that: The method is based on a material pool and an excavator deployed next to the material pool; The excavator includes: a digging arm and two buckets that can open and close to each other; a pressure sensor for detecting the pressure acting on the bucket lips; a camera assembly and a lidar mounted on the base of the excavator facing into the material pool; and a control unit for controlling the digging arm and buckets to dig up the material. The mining method includes the following steps: S1, the position of the wave crest of the material in the material pool and the straight-line distance L1 between the wave crest and the side of the material pool near the excavator are obtained by the camera assembly and lidar; S2, determine whether the straight distance L1 is greater than or equal to the effective digging width D of the bucket. If yes, use the second digging mode to dig the material; otherwise, use the first digging mode to dig the material. The first digging mode is as follows: the bucket is controlled to dig downwards close to the inner wall of the material pool on the side close to the excavator. During this process, the pressure sensor determines whether the bottom has been reached. After the bottom has been reached, the process returns to step S1. The second digging mode is as follows: after adjusting the angle of the buckets so that the bucket lips are both facing the slope on the side of the material closer to the excavator, the two buckets are controlled to insert into the material for digging.

2. The automatic excavation method for excavators based on image recognition according to claim 1, characterized in that: In step S1, the camera assembly and lidar also acquire the slope angle of the material in the area where the wave crest is located; The excavator also includes an angle sensor for detecting the angle of entry of the bucket. The second digging mode also includes adjusting the angle of the buckets according to the slope angle, calibrating the angle sensor so that the bucket lips are all facing the slope of the material closer to the excavator, and then controlling the two buckets to insert into the material for digging.

3. The automatic excavation method for excavators based on image recognition according to claim 1, characterized in that: The camera assembly includes a binocular camera; The camera assembly and lidar acquire the position of the wave crest of the material in the pool, including: The LiDAR scanner scans the material pile within the material pool, generating two-dimensional coordinates (x, y) of all points on the pile surface relative to the excavator. Point cloud preprocessing filters out background and noise points outside the material pool, retaining valid points on the pile surface. Statistical filtering removes isolated points. The material pool plane is divided into 1m×1m grids, and valid points are assigned to their corresponding grids. The average height H(i, j) of each grid and the overall average height Havg of the material pile are calculated. Grids with H(i, j) > Havg are selected from the initial peak regions and clustered into continuous regions. The center coordinates (x1, y1) of each region are taken as the initial peak position, where i represents the row number of the grid in the material pool plane and j represents the column number of the grid. The method for obtaining the position of the wave crest of the material in the pool using the camera assembly and lidar further includes: performing secondary calibration on the initial position of the wave crest using the camera assembly to obtain the final position of the wave crest. The camera component, based on the initial position (x1, y1) output by the LiDAR, verifies and corrects the coordinates using visual features, and outputs the final peak position, including: The camera and LiDAR are synchronized via a time sync device to ensure consistent data acquisition timing. Based on the camera's preset calibration parameters, the initial peak coordinates (x1, y1) of the LiDAR are mapped to the two-dimensional image pixel coordinates (u, v) of the camera, locating the corresponding peak candidate region in the image. A 50×50 pixel sub-region surrounding (u, v) in the image is extracted, and the visual characteristics of this region are analyzed. By using the parallax of the binocular cameras, the relative height gradient of the area is calculated to confirm whether the center position (u,v) is the highest point in the area; if the visual feature verification shows that (x1,y1) is the true highest point of the peak, the coordinates are directly retained. If the visual system detects a shift in the highest point within the region, the shifted two-dimensional coordinates (x2, y2) are deduced through coordinate mapping; the output (x2, y2) is then used as the final position of the peak. Using the excavator's coordinates as the origin (0,0), calculate the straight-line distance L1 based on the final position coordinates (x2,y2) of the wave crest; Methods for obtaining the slope angle of a wave crest from its final position include: Centered on the final position of the wave peak, a rectangular calculation area is formed by extending towards the excavator. The length of the rectangular calculation area is set along the slope extension direction, and the width is set perpendicular to the slope extension direction. From the 3D point cloud of the material pile acquired by the lidar, point clouds whose coordinates fall within the rectangular calculation area are selected to form a local point cloud subset. The local point cloud subset is denoised to remove isolated points and retain effective point clouds that can reflect the surface state of the material pile slope. The denoised effective point clouds are subjected to plane fitting to obtain a fitting plane that matches the surface morphology of the material pile slope. The angle between the fitting plane and the horizontal plane is calculated, and this angle is the slope angle of the corresponding slope area of ​​the material pile. During the excavation process, the camera component transmits excavation images in real time for remote monitoring by personnel.

4. The automatic excavation method for excavators based on image recognition according to claim 3, characterized in that: The two buckets are a first bucket and a second bucket, which together form a bucket assembly. The back of the first bucket is arranged in a straight line, while the back of the second bucket is arranged in an arc shape.

5. The automatic excavation method for excavators based on image recognition according to claim 4, characterized in that: The bucket assembly also includes: A first upper arm and a second upper arm, wherein the first upper arm is mounted on the first bucket and the second upper arm is mounted on the second bucket, and the first upper arm and the second upper arm are hinged to each other; The support frame has a hinge point where the first upper arm and the second upper arm are hinged to each other, which is rotatably connected to the inner side of the support frame via a rotating shaft. A first hydraulic cylinder and a second hydraulic cylinder, wherein the cylinder end of the first hydraulic cylinder is hinged to the inner top side of the support frame, and the telescopic end is hinged to the upper end of the first upper arm; the cylinder end of the second hydraulic cylinder is hinged to the inner top side of the support frame, and the telescopic end is hinged to the upper end of the second upper arm; and two pressure sensors are provided, which are respectively installed at the pressure measuring ports of the first hydraulic cylinder and the second hydraulic cylinder. A connecting frame, the front end of which is rotatably connected to the support frame, and the rear end of which is rotatably connected to the excavator's boom; The third hydraulic cylinder has its cylinder end hinged to the inner side of the top of the connecting frame, and its telescopic end hinged to the outer side of the support frame. The control unit is connected to and controls the excavator arm, the first hydraulic cylinder, the second hydraulic cylinder, and the third hydraulic cylinder. The control unit drives the opening and closing of the first bucket and the second bucket by controlling the extension and retraction of the first hydraulic cylinder and / or the second hydraulic cylinder. The control unit adjusts the orientation of the first bucket and the second bucket by controlling the extension and retraction of the third hydraulic cylinder.

6. The automatic excavation method for excavators based on image recognition according to claim 5, characterized in that: The first mining mode includes the following steps: S11. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to open the first bucket and the second bucket to the maximum angle, adjusts the action of the digging arm so that the back of the first bucket is in close contact with the inner wall of the material pool, and drives the entire bucket assembly to move vertically downward so that the first bucket and the second bucket are inserted into the material pile. S12. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to close the first bucket and the second bucket together, and then controls the excavator's arm to pull out the bucket assembly to complete the digging. In step S11, when the bucket assembly moves downward, if it is determined that the bucket has touched the bottom, the downward movement of the bucket is stopped. Then, in step S12, during the process of the first bucket and the second bucket closing with each other, the bucket lips of the first bucket and the second bucket are ensured to be in contact with the inner bottom surface of the material pool.

7. The automatic excavation method for excavators based on image recognition according to claim 5, characterized in that: The second mining mode includes the following steps: S21. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to open the first bucket and the second bucket to the maximum angle, and adjusts the angle of the bucket assembly by the third hydraulic cylinder so that the bucket lips of the first bucket and the second bucket face the slope closer to the excavator. The control unit controls the arm so that the bucket lips of the first bucket and the second bucket are inserted into the slope facing the slope closer to the excavator. S22. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to close the first bucket and the second bucket together, and then controls the excavator's arm to pull out the bucket assembly to complete the digging.

8. The automatic excavation method for excavators based on image recognition according to claim 7, characterized in that: The laser radar defines the wave peak as an area with a relative height ≥ 0.5m or a slope angle > 30°. The relative height is the difference between the overall average height Havg of the stockpile and the height of the target stockpile. The excavation method further includes: if no wave peak is detected, for areas outside the effective excavation width D of the bucket, the following steps are used for excavation: S311. The control unit controls the first hydraulic cylinder and / or the second hydraulic cylinder to open the first bucket and the second bucket to the maximum angle. The angle of the bucket assembly is adjusted by the extension and retraction of the third hydraulic cylinder so that the bucket lips of the first bucket and the second bucket face the surface of the material pile. The control unit controls the digging arm so that the bucket lips of the first bucket and the second bucket are inserted into the material pile at a vertical angle. S312. The control unit controls the first bucket and the second bucket to close together, and then controls the excavator's arm to pull out the bucket assembly to complete the digging. For areas within the effective digging width D of the bucket, the first digging mode is used for digging.

Images