Intelligent leveling control method and system for engineering machinery and engineering machinery

Abstract

The invention relates to an intelligent leveling control method for engineering machinery. The method comprises the following steps: acquiring image data including depth information of an operation object of the engineering machinery through a camera device; performing 3D modeling on the operation object based on the image data from the camera device to obtain the current shape of the operation object; determining a difference between the current shape and an expected shape of the work object; and controlling an operation of the construction machine based on the difference so that the shape of the work object matches the desired shape. The invention further relates to an intelligent leveling control system for the engineering machinery and the engineering machinery comprising the control system.

Description

Technical Field

[0001] The present invention relates to the technical field of engineering machinery, and more specifically to an intelligent leveling control method and control system for engineering machinery based on image recognition, and an engineering machinery comprising the control system. Background Art

[0002] Many types of construction machinery are equipped with blade systems. For example, a motor grader is a widely used type of construction machinery. It primarily uses its blade to cut, spread, or level the ground. It can be used for various construction tasks, including building roadbeds and pavements, constructing slopes, digging ditches, clearing snow, and mixing and moving loose materials.

[0003] During the operation of construction machinery, it is necessary to accurately detect and control the position and posture of the blade or other actuators. In order to reduce the operator's operating difficulty and workload, some intelligent control systems have been developed to automatically control the blade posture.

[0004] Current intelligent leveling systems (Grade Control Systems) for construction machinery are primarily based on sensors and GPS, utilizing RTK differential positioning principles to position and control the actuators of construction machinery. However, this current technical solution relies on mobile base stations, and GPS signals are susceptible to interference, making it unusable in underground environments and mining operations. Furthermore, the absolute accuracy of positioning and control is limited, typically around + / - 2cm. Furthermore, RTK differential positioning is costly, and the corresponding control system is complex to install, requiring extensive system integration work.

[0005] The present invention is directed to solving at least one of the above-mentioned problems of the prior art as well as other problems. Summary of the Invention

[0006] According to one aspect of the present invention, a method for intelligent leveling control of construction machinery is proposed, comprising:

[0007] Acquiring image data including depth information of a working object of the engineering machinery through a camera device;

[0008] Performing 3D modeling on the work object based on the image data from the camera device to obtain the current shape of the work object;

[0009] determining a difference between the current shape and a desired shape of the work object; and

[0010] The operation of the construction machine is controlled based on the difference so that the shape of the work object matches the desired shape.

[0011] Advantageously, the depth information is generated based on a plurality of images acquired by a camera device.

[0012] Advantageously, a working posture of the actuator of the construction machine and a travel path of the construction machine are calculated based on the difference.

[0013] Advantageously, performing 3D modeling on the work object based on the image data from the camera device to obtain the current shape of the work object comprises the following steps:

[0014] Updating image data including depth information based on the camera's pose;

[0015] The current image data including depth information is fused with the established 3D model to obtain the latest 3D model.

[0016] Advantageously, the control method includes displaying the current shape and the desired shape of the work object on a display in real time.

[0017] According to another aspect of the present invention, an intelligent leveling control system for construction machinery is provided, comprising:

[0018] a camera device for acquiring image data including depth information of a working object of the engineering machinery; and

[0019] A controller is configured to implement the control method according to the present invention.

[0020] Advantageously, the control system further comprises a display for displaying the current shape and the desired shape of the work object in real time.

[0021] Advantageously, the controller is integrated into the display.

[0022] Advantageously, the control system further comprises a first solenoid valve group and a second solenoid valve group respectively connected to the controller signal, the first solenoid valve group being connected to the hydraulic cylinder fluid of the actuator of the engineering machinery, and the second solenoid valve group being connected to the steering system fluid of the engineering machinery.

[0023] Advantageously, the first solenoid valve group and the second solenoid valve group each include a plurality of proportional solenoid valves assembled into a valve block.

[0024] Advantageously, the camera arrangement comprises two or more cameras.

[0025] Advantageously, the camera device is two binocular cameras.

[0026] Advantageously, the controller comprises a GPU.

[0027] According to another aspect of the present invention, a construction machine is provided, which includes the control system according to the present invention.

[0028] Advantageously, the construction machine is a motor grader.

[0029] The main advantages of the control method and control system according to the present invention include:

[0030] (1) It can model the working objects of engineering machinery in real time, achieve a ranging accuracy of several millimeters or even higher, improve the control accuracy of engineering machinery, and reduce costs;

[0031] (2) No base station is required, suitable for multiple scenarios, especially for use in underground spaces and mine tunnels;

[0032] (3) It can automatically control the operation of the actuators of the construction machinery and the travel route of the construction machinery, greatly improving the operating efficiency of the construction machinery;

[0033] (4) Presenting the 3D real scene of the work object in real time through the display;

[0034] (5) The control system is constructed as an independent system including monitoring functions and execution functions. It is independent of the control system of the entire vehicle and can be easily installed in existing engineering machinery. It can also switch between the control system of the present invention and the original control system of the engineering machinery, so that the operator can select the appropriate control system according to needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying schematic drawings. The accompanying drawings and corresponding embodiments are for illustrative purposes only and are not intended to limit the present invention. In the accompanying drawings:

[0036] Figure 1 It is a schematic flow chart of an intelligent leveling control method for engineering machinery according to a preferred embodiment of the present invention.

[0037] Figure 2 It is a schematic architecture diagram of an intelligent leveling control system for engineering machinery according to a preferred embodiment of the present invention.

[0038] List of reference numerals:

[0039] 1 Camera 2 Controller

[0040] 3 Display 4 Graphics Processing Unit

[0041] 8 First solenoid valve group 9 Second solenoid valve group

[0042] 11 Actuator 12 Steering system

[0043] 100 control systems DETAILED DESCRIPTION

[0044] Embodiments of the present invention are described below with reference to the accompanying drawings. In the following description, many specific details are set forth to enable those skilled in the art to more fully understand and implement the present invention. However, it will be apparent to those skilled in the art that the present invention may be implemented without some of these specific details. Furthermore, it should be understood that the present invention is not limited to the specific embodiments described. Rather, it is contemplated that the present invention may be implemented with any combination of the features and elements described below, regardless of whether they relate to different embodiments. Therefore, the aspects, features, embodiments, and advantages described below are intended for illustrative purposes only and should not be considered as elements or limitations of the claims unless expressly set forth in the claims.

[0045] Figure 1 The figure is a schematic flow chart of an intelligent leveling control method for construction machinery according to a preferred embodiment of the present invention. The construction machinery may be a grader, excavator, bulldozer, compactor, paver, scraper, or other type of machinery with similar working conditions.

[0046] The control method according to the present invention comprises the following steps:

[0047] S1: Acquire image data including depth information of a working object of the construction machinery through a camera device;

[0048] S2: Performing 3D modeling on the work object based on the image data from the camera device to obtain the current shape of the work object;

[0049] S3: determining a difference between the current shape and a desired shape of the work object; and

[0050] S4: Controlling the operation of the construction machine based on the difference so that the shape of the work object matches the desired shape.

[0051] Advantageously, the control method generates the depth information based on a plurality of images acquired by a camera device.

[0052] Advantageously, the control method calculates the working posture of the construction machinery's actuator and the construction machinery's travel path based on the differences, thereby automatically controlling the operation of the construction machinery. This control method can significantly improve the operating efficiency of the construction machinery.

[0053] For example, in an application where the construction machinery is a motor grader, this control method, after creating a 3D model of the motor grader's working surface (i.e., the work object), can then use an adaptive algorithm to calculate the working posture of the motor grader's blade at different locations, including operating parameters such as height, cutting angle, and side shift position, based on the difference between the current working surface shape and the desired working surface shape. It can also use an appropriate path planning algorithm to plan the motor grader's path, thereby achieving automatic control of the motor grader. The path planning algorithm here can be, for example, the D* algorithm, an incremental search algorithm suitable for path planning within a known map and capable of handling environmental changes at runtime. It dynamically updates the path based on a cost map and a heuristic function, and makes corrections to adapt to environmental changes.

[0054] Advantageously, step S2 comprises the following steps:

[0055] S21: updating the image data including the depth information based on the camera's position;

[0056] S22: Fusing the current image data including the depth information with the established 3D model to obtain the latest 3D model.

[0057] During the operation of construction machinery, the camera's pose (position and attitude) constantly changes. Camera tracking can eliminate the accumulated error in pose estimation, ultimately obtaining a dense surface representation such as the latest 3D model and realizing three-dimensional reconstruction of dynamic scenes based on stereo representation.

[0058] Advantageously, step S2 may further include performing at least one of the following processing on the image data: noise reduction, color enhancement, and texture enhancement.

[0059] For example, step S2 may include a pre-processing stage, in which bilateral filtering is performed on the image data to remove noise contained in the image data, and the image data is converted into a dense vertex map and a normal vector map.

[0060] In addition, by adding color and / or texture information to the model, the reconstructed 3D model can be made more realistic and achieve better visual effects.

[0061] Advantageously, a TSDF (Truncated Signed Distance Function) model can be used for 3D reconstruction. It is understood that other known 3D reconstruction algorithms can also be used to perform 3D modeling of the work object.

[0062] The control method according to the present invention utilizes RGB-D (color image + depth image) to model the working object of engineering machinery in real time, which can achieve a ranging accuracy of several millimeters or even higher, thereby improving the control accuracy of engineering machinery and reducing costs.

[0063] Figure 2 1 is a schematic architecture diagram of an intelligent leveling control system 100 for construction machinery according to a preferred embodiment of the present invention.

[0064] like Figure 2 As shown, the control system 100 includes a camera 1, which is used to obtain image data including depth information of a working object of an engineering machine. The camera 1 may include two or more cameras. The camera may be a depth camera, such as a binocular camera.

[0065] In one embodiment, the camera device 1 includes two cameras mounted on either side of the front end of the construction machinery. These two cameras simultaneously observe the same scene (i.e., the working object of the construction machinery, such as the work surface in front of the construction machinery) and infer the scene's depth information by calculating the parallax between the two cameras. By using a deep neural network, a point cloud or depth image can be directly predicted from the camera image, enabling fast and accurate 3D scanning and modeling of the work surface to be operated. Commonly used algorithms include stereo matching algorithms (such as SAD, SSD, SIFT, etc.) and disparity map optimization algorithms (such as SGBM, SGM, etc.).

[0066] It is understandable that the more cameras used, the more comprehensive the information collected. The specific number of cameras can be selected based on factors such as cost and computational complexity.

[0067] Control system 100 also includes a controller 2, which is signal-connected to camera 1 and capable of receiving image data captured by camera 1. Controller 2 is configured to implement the control method according to the present invention. Controller 2 advantageously includes a GPU (graphics processing unit) 4. The powerful computing power of the GPU enables the aforementioned rapid and accurate 3D scanning and modeling from camera images using deep neural networks.

[0068] Specifically, the controller 2 is configured to:

[0069] Performing 3D modeling on the work object based on the image data from the camera device 1 to obtain the current shape of the work object;

[0070] determining a difference between the current shape and a desired shape of the work object; and

[0071] The operation of the construction machine is controlled based on the difference so that the shape of the work object matches the desired shape.

[0072] Advantageously, the controller 2 is configured to calculate the working posture of the actuator of the engineering machine based on the difference by an adaptive algorithm, and to calculate the travel path of the engineering machine by adopting a suitable path planning algorithm.

[0073] Advantageously, the controller 2 performs 3D modeling on the work object based on the image data from the camera 1 to obtain the current shape of the work object, including the following steps:

[0074] Updating image data including depth information based on the camera's pose;

[0075] The current image data including depth information is fused with the established 3D model to obtain the latest 3D model.

[0076] Advantageously, data describing the desired shape of the work object may be input into a memory (not shown) of the controller 2 in advance, and the controller 2 can read the data from the memory when needed.

[0077] The control system 100 further includes a display 3, which can display the current shape and the desired shape of the work object in real time on the display 3. The display 3 can be set in the cab of the engineering machine, or set at other locations of the engineering machine, or set at a location away from the engineering machine.

[0078] The controller 2 may be integrated into the display 3 or may be a component separate from the display 3. Preferably, the GPU 4 is integrated into the display 3.

[0079] like Figure 2 As shown, the control system 100 also includes a first solenoid valve group 8 and a second solenoid valve group 9 respectively connected to the controller 2 signal. The first solenoid valve group 8 is connected to the hydraulic cylinder fluid of the actuator 11 for the engineering machinery, aiming to achieve control of the shovel by controlling the hydraulic cylinder. The second solenoid valve group 9 is connected to the steering system 12 fluid of the engineering machinery.

[0080] After controller 2 calculates the working posture of the construction machine's actuator 11 and the construction machine's travel path based on the image data from camera device 1, controller 2 sends control signals related to the working posture of actuator 11 to first solenoid valve group 8 in real time to control the operation of actuator 11, and sends control signals related to the construction machine's travel path to second solenoid valve group 9 in real time to control the operation of steering system 12. Simultaneously, display 3 can provide real-time feedback on the working posture of actuator 11 and the position of the construction machine, achieving closed-loop control.

[0081] Advantageously, the first solenoid valve group 8 and the second solenoid valve group 9 each include a plurality of proportional solenoid valves assembled into a valve block.

[0082] The control system 100 according to the present invention is constructed as an independent system including monitoring functions and execution functions. It is independent of the control system of the entire vehicle and can be easily installed in existing engineering machinery. It can also switch between the control system 100 and the original control system of the engineering machinery, so that the operator can select a suitable control system according to needs.

[0083] The control system 100 does not require a base station and is suitable for multiple scenarios, especially underground spaces and mine tunnels. Furthermore, the control system 100 is applicable to various types of construction machinery, such as, but not limited to, graders, excavators, bulldozers, compactors, pavers, and scrapers.

[0084] The intelligent leveling control method and control system for engineering machinery of the present invention are described above with the aid of specific embodiments. It will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments disclosed above without departing from the scope or spirit of the invention. For example, the implementation of the present invention may not include some of the specific features described, and the present invention is not limited to the specific embodiments described, but rather any combination of the features and elements described may be envisioned. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed control method and control system. The description and examples are to be regarded as exemplary only, with the true scope being indicated by the appended claims and their equivalents.

Claims

1. An intelligent leveling control method for construction machinery, comprising: Acquiring image data including depth information of a working object of the engineering machinery through a camera device; Performing 3D modeling on the work object based on the image data from the camera device to obtain a current shape of the work object; determining a difference between the current shape and a desired shape of the work object; as well as The operation of the construction machine is controlled based on the difference so that the shape of the work object matches the desired shape.

2. The control method according to claim 1, wherein: The depth information is generated based on a plurality of images acquired by the camera device.

3. The control method according to claim 1 or 2, wherein: Based on the difference, the working posture of the actuator of the construction machine and the travel path of the construction machine are calculated.

4. The control method according to claim 1 or 2, wherein: Performing 3D modeling of the work object based on the image data from the camera device to obtain the current shape of the work object comprises the following steps: Updating image data including depth information based on the camera's pose; The current image data including depth information is fused with the established 3D model to obtain the latest 3D model.

5. The control method according to claim 1 or 2, wherein: The control method includes displaying a current shape and a desired shape of a work object on a display in real time.

6. An intelligent leveling control system for construction machinery, comprising: a camera device for acquiring image data including depth information of a working object of the engineering machinery; and A controller configured to implement the control method according to any one of claims 1 to 5.

7. The control system according to claim 6, wherein: The control system further includes a display configured to display a current shape and a desired shape of the work object in real time.

8. The control system according to claim 7, wherein: The controller is integrated into the display.

9. The control system according to claim 6 or 7, wherein: The control system further includes a first solenoid valve group and a second solenoid valve group respectively connected to controller signals, the first solenoid valve group being fluidly connected to a hydraulic cylinder of an actuator of the engineering machinery, and the second solenoid valve group being fluidly connected to a steering system of the engineering machinery.

10. The control system according to claim 9, wherein: The first solenoid valve group and the second solenoid valve group each include a plurality of proportional solenoid valves assembled into a valve block.

11. The control system according to claim 6, wherein: The imaging device includes two or more cameras.

12. The control system according to claim 11, wherein: The camera device is two binocular cameras.

13. The control system according to claim 6, wherein: The controller includes a GPU.

14. A construction machine comprising the control system according to any one of claims 6 to 13.

15. The construction machine according to claim 14, wherein: The construction machine is a motor grader.

Images