Complete excavator control system based on machine vision and excavator

Abstract

The invention discloses an excavator complete machine control system based on machine vision and an excavator, the excavator complete machine control system comprises an AI camera and a vehicle control unit, and the AI camera and the vehicle control unit interact through a CAN bus; the AI camera is used for acquiring an operation image, the operation image is identified through an image processor embedded in the AI camera to obtain a working condition and an action, and the AI camera sends the obtained working condition and action to the vehicle control unit through a CAN bus; and matching data are pre-stored in the vehicle control unit, and the matching data are called according to the received current working condition and action and parameter adjustment is carried out. Compared with traditional pump pressure recognition, the method has the advantages that action response delay can be reduced, secondary operation caused by weak excavation is avoided, similar actions can be effectively distinguished through fuzzy recognition, the parameter matching rate is increased, and oil consumption waste or action abnormity is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of excavator whole-machine control, and in particular to an excavator whole-machine control system and an excavator based on machine vision. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Among construction machinery, excavator operating efficiency and energy consumption control have always been core concerns for the industry. With the development of intelligent technology, achieving accurate identification of complex working conditions and adaptive parameter adjustment has become a key breakthrough in improving overall machine performance.

[0004] The motion recognition and parameter debugging of excavators traditionally rely on data collected by the main pump pressure sensor and the pilot pressure sensor. By analyzing the pump pressure and handle motion signals, the operating status (such as excavation, leveling, etc.) is determined and the parameters of the engine and hydraulic system are adjusted accordingly.

[0005] In existing technology, pressure sensors are used to identify actions. Pump pressure data is collected by the main pump pressure sensor, and combined with a pilot pressure sensor to detect operator handle movements (such as the pilot pressure signal for boom raising and digging with the dipper arm), this determines the current action. Based on load feedback after pump pressure is established, parameters such as engine speed and hydraulic pump displacement are passively adjusted to suit the task. However, research has found that the pump pressure signal is directly related to the load. Load changes are only detected after a heavy load is applied and pump pressure is established. At this point, increasing digging force is too late, and there's no way to reserve power in advance. This leads to delayed recognition and response, resulting in low operational efficiency. Furthermore, different actions (such as digging and grading) may generate similar pilot pressure signals, but require significantly different hydraulic displacements. Relying solely on pressure sensors cannot accurately distinguish the actual action, which can easily lead to mismatched parameters, resulting in wasted fuel or abnormal operation (such as nodding during grading). Summary of the Invention

[0006] In order to solve the above problems, the present invention proposes an excavator control system and excavator based on machine vision, which uses AI cameras and vehicle controllers to communicate and adjust vehicle parameters to achieve accurate identification and rapid response.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a complete excavator control system based on machine vision, comprising: An AI camera and a vehicle controller, wherein the AI ​​camera and the vehicle controller interact via a CAN bus; The AI ​​camera is used to obtain an image of the excavator's operation, identify the image of the excavator's operation through an internally embedded image processor, and obtain the working condition and action; set parameter data according to the working condition and action, and pre-store the parameter data in the vehicle controller; the AI ​​camera sends the obtained working condition and action to the vehicle controller in the form of a unique code ID via the CAN bus; the vehicle controller retrieves the corresponding parameter data based on the received unique code ID and adjusts the parameters; After the excavator has been digging continuously, the AI ​​camera also sends an abnormal state ID to the vehicle controller through the CAN bus. After receiving the state ID, the vehicle controller converts it into a linear adjustment parameter. The linear adjustment parameter is selected based on the difference between the pre-calibrated ideal state parameter and the current abnormal state parameter, and the step size to be adjusted is selected.

[0008] According to a further technical solution, the image processor recognizes the excavator operation image and obtains the working conditions including earthwork, earth and stonework, stonework and flat ground.

[0009] A further technical solution is that under the earthwork working condition, the vehicle controller calls the group one data in the parameter data to adjust the parameters. When adjusting the parameters through the group one data, the group one data is to reduce fuel consumption and adjust the power and speed to the best; under the stonework working condition, the vehicle controller calls the group three data in the parameter data to adjust the parameters. When adjusting the parameters through the group three data, the group three data is to increase the power and speed and reduce the speed regulation or no speed regulation.

[0010] A further technical solution is that under the earthwork working condition, the vehicle controller retrieves the second group of parameter data to adjust the parameters, and the adjustment strategy is between the earthwork working condition and the rockwork working condition.

[0011] According to a further technical solution, the actions obtained after the image processor recognizes the excavator operation image include four parts, namely the excavation stage, the loading stage, the unloading stage and the return stage.

[0012] A further technical solution is that the action in the excavation stage is digging by the dipper arm plus digging by the bucket and partial lifting of the boom, the action in the loading stage is lifting of the boom plus rotation, the action in the unloading stage is unloading by the dipper arm plus unloading by the bucket, and the action in the return stage is rotation plus lowering of the boom.

[0013] According to a further technical solution, the vehicle controller retrieves group 1 data from the matching data for parameter adjustment. When adjusting parameters through group 1 data, group 1 data is used to adjust power and speed to the best with the goal of reducing fuel consumption; under the rockwork condition, the vehicle controller retrieves group 3 data from the matching data for parameter adjustment. When adjusting parameters through group 3 data, group 3 data is used to increase power and speed and reduce rate regulation or no rate regulation; under the earthwork condition, the vehicle controller retrieves group 2 data from the matching data for parameter adjustment, and the adjustment strategy is between the earthwork condition and the rockwork condition.

[0014] As a further technical solution, during the excavation phase, the AI ​​camera sends the state of the excavator at the next moment to the vehicle controller at a fixed period, and the vehicle controller adjusts the parameter values ​​to the optimal excavation parameters.

[0015] For a further technical solution, the AI ​​camera obtains the rotation angle and boom lifting height and sends them to the vehicle controller. After receiving them, the vehicle controller adjusts the proportional parameters of the PNS valve.

[0016] As a further technical solution, the vehicle controller adjusts the proportional parameters of the PNS valve to supply hydraulic oil to the rotation and reduce power to ensure the return position and fuel consumption.

[0017] In a second aspect, the present invention provides an excavator, comprising: an excavator control system based on machine vision as described in any one of the first aspects.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention uses an AI camera to capture working images, an image processor to identify working conditions and actions, and transmits these to the vehicle controller, adjusting parameters in advance to ensure that digging force is fully reserved before contact with the load. Compared with the "after-the-fact adjustment" of traditional pump pressure recognition, this method can reduce action response delays, avoid secondary operations caused by insufficient digging force, and improve work efficiency. Furthermore, image processor recognition avoids the "fuzzy recognition" of pressure sensors, effectively distinguishing similar actions, improving parameter matching rates, and avoiding fuel waste or abnormal operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0020] Figure 1 This is a control logic diagram of a whole machine control system based on machine vision according to the present invention; DETAILED DESCRIPTION The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0022] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0023] Example 1 This embodiment provides an excavator control system based on machine vision. The control system includes an AI camera and a vehicle controller MCU. An image processor is embedded in the AI ​​camera, and the image processor in the AI ​​camera interacts with the vehicle controller MCU through a CAN bus.

[0024] In this embodiment, two AI cameras are provided, and the two AI cameras are installed at the uppermost ends of the excavator cab respectively, for acquiring and storing the working images of the excavator. The two AI cameras serve to complement and verify each other. After the AI ​​camera acquires the working image, it uses the image processor to identify the working image to obtain the working condition and action. In this embodiment, the working conditions obtained after the image processor identifies the working image include earthwork, earthwork, stonework and flat ground. The loading and dumping actions obtained after the image processor identifies the working image include four parts, namely the excavation stage, the loading stage, the unloading stage and the return stage. The action of the excavation stage is the digging of the bucket arm plus the digging of the bucket and the lifting of the partial boom. The action of the loading stage is the lifting of the boom plus the rotation. The action of the unloading stage is the unloading of the bucket arm plus the unloading of the bucket. The action of the return stage is the rotation plus the lowering of the boom. The difference between loading and dumping is that the height of the loading boom is higher and the digging of the bucket arm is farther.

[0025] In this embodiment, the image processor uses a conventional neural network model to recognize the task image. As long as the same goal of action recognition in the image can be achieved, it will not be described in detail in this embodiment.

[0026] like Figure 1 As shown in the figure, the AI ​​camera sends the obtained working conditions and actions to the vehicle controller through the CAN bus, informing the vehicle controller of the current working conditions and actions of the excavator. The vehicle controller has pre-stored matching data. According to the current working conditions and actions received, the matching data is retrieved and parameters are adjusted. The specific parameter adjustment method is as follows: When it is identified that the excavator is in earthwork condition, which is light load operation, more emphasis is placed on fuel consumption and driving comfort. The vehicle controller calls group one data in the matching data for parameter adjustment. When adjusting parameters through group one data, group one data is to reduce fuel consumption and adjust power and speed to the best; when it is identified that the excavator is in rockwork condition, which is heavy load operation, more emphasis is placed on work efficiency. The vehicle controller calls group three data in the matching data for parameter adjustment. When adjusting parameters through group three data, group three data is to increase power and speed and reduce speed regulation or no speed regulation to ensure digging force, etc.; when it is identified that the excavator is in earthwork condition, the load is between earthwork and rockwork, the vehicle controller calls group two data in the matching data for parameter adjustment, and the adjustment strategy is between earthwork condition and rockwork condition.

[0027] During the operation, the power required in the excavation phase is relatively large. When it is recognized that the bucket is about to enter the soil, the AI ​​camera sends the state of the excavator at the next moment to the vehicle controller at a fixed period. The vehicle controller adjusts the parameter value to the optimal excavation parameter. This period cannot be too long, which will cause loading delays. In this embodiment, the fixed period is preferably 10ms. During the loading phase, the AI ​​camera continuously obtains the rotation angle and boom lifting height and sends them to the vehicle controller. After receiving them, the vehicle controller adjusts the proportional parameters of the PNS valve to ensure the matching of the rotation and boom movements. During the return phase, the boom descends and rotates. The vehicle controller adjusts the proportional parameters of the PNS valve to supply hydraulic oil to the rotation and reduce power to ensure the return position and fuel consumption.

[0028] In leveling conditions, the boom is raised and the dipper arm is used for digging, similar to the excavation phase. However, leveling requires less power and a lower arm displacement than digging, otherwise fuel consumption and digging problems may occur. While the vehicle controller cannot clearly distinguish between the two actions, the AI ​​camera can identify them. When it detects leveling conditions, it notifies the MCU via ID_X, which then proactively adjusts the parameters and controls the vehicle's hydraulic system for leveling.

[0029] Specifically: The boom excavation valve core typically moves quickly and is difficult to control due to its regenerative nature. The heavy weight of the boom and high pump pressure during boom raising result in less flow to the boom. Therefore, adjustments are primarily made to the flow distribution and flow loading process.

[0030] During the leveling phase, the boom pump's flow rate is actively reduced, and the flow loading rate of the corresponding boom pump is limited (via current loading rate control). The boom pump's flow distribution and loading rate are then increased. This allows the boom to rise faster initially, avoiding initial nodding issues. During the next stage, the boom flow rate is increased and the boom flow rate is reduced for more coordinated movement.

[0031] Excavation phase: Compared to flat ground, this phase requires stronger digging force from the arm and bucket, resulting in a larger displacement. Larger displacement corresponds to greater torque, increasing digging force. Therefore, the displacement of the arm pump is increased. If the total power of the two pumps is 2, the power of the main arm pump can be appropriately allocated to 1.1-1.2. This allows for further increase in displacement even when constant power is reached, i.e., when digging heavy objects.

[0032] In this embodiment, the visual instrument will adjust and learn over a long period of time. For example, when digging becomes weak after 10 or more consecutive digging operations, the AI ​​camera will send an abnormal state ID to the vehicle controller through the CAN bus. After receiving the state ID, the vehicle controller will adjust the parameters linearly. The linear adjustment parameters are the step size to be adjusted based on the difference between the pre-calibrated ideal state parameters (maximum and minimum value range) and the current abnormal state parameters, such as incremental PID adjustment.

[0033] Example 2 This embodiment provides an excavator, including: an excavator control system based on machine vision as provided in the first embodiment.

[0034] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

[0035] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A complete excavator control system based on machine vision, characterized in that: include: An AI camera and a vehicle controller, wherein the AI ​​camera and the vehicle controller interact via a CAN bus; The AI ​​camera is used to obtain an image of the excavator's operation, identify the image of the excavator's operation through an internally embedded image processor, and obtain the working condition and action; set parameter data according to the working condition and action, and pre-store the parameter data in the vehicle controller; the AI ​​camera sends the obtained working condition and action to the vehicle controller in the form of a unique code ID via the CAN bus; the vehicle controller retrieves the corresponding parameter data based on the received unique code ID and adjusts the parameters; After the excavator has been digging continuously, the AI ​​camera also sends an abnormal state ID to the vehicle controller through the CAN bus. After receiving the state ID, the vehicle controller converts it into a linear adjustment parameter. The linear adjustment parameter is selected based on the difference between the pre-calibrated ideal state parameter and the current abnormal state parameter, and the step size to be adjusted is selected.

2. The excavator control system based on machine vision according to claim 1, characterized in that: The working conditions obtained after the image processor recognizes the excavator operation image include earthwork, earth and stonework, stonework and level ground.

3. The machine vision-based excavator control system according to claim 2, characterized in that: Under the earthwork working condition, the vehicle controller calls group 1 data from the parameter data for parameter adjustment. When adjusting the parameters through group 1 data, group 1 data is to reduce fuel consumption and adjust the power and speed to the best; under the stonework working condition, the vehicle controller calls group 3 data from the parameter data for parameter adjustment. When adjusting the parameters through group 3 data, group 3 data is to increase power and speed and reduce the speed regulation or no speed regulation.

4. The machine vision-based excavator control system according to claim 2, characterized in that: Under the earthwork working condition, the vehicle controller retrieves the second group of parameter data to adjust the parameters, and the adjustment strategy is between the earthwork working condition and the rockwork working condition.

5. The excavator control system based on machine vision according to claim 1, characterized in that: The actions obtained after the image processor recognizes the excavator operation image include four parts, namely, the excavation stage, the loading stage, the unloading stage and the return stage.

6. The machine vision-based excavator control system according to claim 5, characterized in that: The action in the excavation stage is arm excavation plus bucket excavation and partial boom lifting, the action in the loading stage is boom lifting plus rotation, the action in the unloading stage is arm unloading plus bucket unloading, and the action in the return stage is rotation plus boom lowering.

7. The machine vision-based excavator control system according to claim 5, characterized in that: During the excavation phase, the AI ​​camera sends the excavator status at the next moment to the vehicle controller at a fixed period, and the vehicle controller adjusts the parameter values ​​to the optimal excavation parameters.

8. The machine vision-based excavator control system according to claim 5, characterized in that: During the loading stage, the AI ​​camera obtains the rotation angle and boom lifting height and sends them to the vehicle controller. After receiving them, the vehicle controller adjusts the proportional parameters of the PNS valve.

9. The machine vision-based excavator control system according to claim 5, characterized in that: During the return phase, the vehicle controller adjusts the proportional parameters of the PNS valve to supply hydraulic oil to the rotation and reduce power to ensure the return position and oil consumption.

10. An excavator, characterized in that: include: A complete excavator control system based on machine vision as described in any one of claims 1 to 9.

Images