A machine vision-based whole machine control system of an excavator and the excavator

Abstract

The application discloses a kind of based on machine vision's excavator whole machine control system and excavator, comprising: AI camera and whole vehicle controller, the AI camera with whole vehicle controller is interacted by CAN bus;AI camera is used to obtain work image, and work image is identified by internal embedded image processor, and working condition and action are obtained, the AI camera is sent to the whole vehicle controller by CAN bus with the working condition and action obtained;Matching data is stored in the whole vehicle controller in advance, and matching data is retrieved and parameter adjustment according to the current working condition and action received.The application compared with traditional pump pressure identification, can reduce action response delay, avoid the secondary operation caused by excavating powerlessness, also avoid "fuzzy recognition" can effectively distinguish similar action, improve the matching rate of parameter, avoid causing oil consumption waste or action anomaly.

Description

Technical Field

[0001] This invention relates to the field of excavator control technology, and in particular to an excavator control system and excavator based on machine vision. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In the construction machinery industry, the operating efficiency and energy consumption control of excavators have always been core issues of concern. With the development of intelligent technology, how to achieve accurate identification of complex working conditions and adaptive parameter adjustment has become a key breakthrough in improving the overall performance of the machine.

[0004] Traditionally, excavator motion recognition and parameter adjustment rely on data collected by the main pump pressure sensor and pilot pressure sensor. The working status (such as digging, leveling, etc.) is determined by analyzing the pump pressure and handle action signals, and the parameters of the engine and hydraulic system are adjusted accordingly.

[0005] In existing technologies, pressure sensors are used to primarily identify actions. A main pump pressure sensor collects pump pressure data, which is combined with a pilot pressure sensor to detect operator handle movements (such as pilot pressure signals for boom lifting and stick digging). This determines the current operational action, and based on load feedback after pump pressure is established, parameters such as engine speed and hydraulic pump displacement are passively adjusted to adapt to operational needs. However, research has revealed that the pump pressure signal is directly related to the load. Load changes are only detected after heavy load is encountered and pump pressure is established. By this time, increasing digging force is "too late," as power cannot be reserved in advance, resulting in recognition lag and response delay, leading to low operational efficiency. Furthermore, different operational actions (such as digging and leveling) may generate similar pilot pressure signals, but the required hydraulic displacement differs significantly. Relying solely on pressure sensors cannot accurately distinguish actual actions, easily leading to improper parameter matching, resulting in wasted fuel or abnormal actions (such as nodding during leveling). Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a machine vision-based excavator control system and excavator, which utilizes an AI camera and a vehicle controller to communicate and adjust vehicle parameters, achieving accurate identification and rapid response.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a machine vision-based excavator control system, comprising: The AI ​​camera and the vehicle controller interact via a CAN bus; The AI ​​camera is used to acquire images of the excavator in operation. An internally embedded image processor identifies these images to determine the operating conditions and actions. Based on these conditions and actions, parameter data is set and pre-stored in the vehicle controller. The AI ​​camera then transmits the acquired operating conditions and actions to the vehicle controller via the CAN bus in the form of a unique coded ID. The vehicle controller retrieves the corresponding parameter data based on the received unique coded ID and adjusts the parameters accordingly. After the excavator has been digging continuously, the AI ​​camera also sends an abnormal status ID to the vehicle controller via the CAN bus. After receiving the status ID, the vehicle controller adjusts the parameters linearly. The linear adjustment parameters are selected based on the difference between the pre-calibrated ideal status parameters and the current abnormal status parameters, and the adjustment step size is selected accordingly.

[0008] A further technical solution involves using the image processor to identify the working conditions of the excavator, which include earthwork, earth and rock excavation, rock excavation, and leveling.

[0009] A further technical solution is that, under the earthmoving condition, the vehicle controller retrieves the first group of parameter data for parameter adjustment. When adjusting parameters using the first group of data, the first group of data aims to reduce fuel consumption by adjusting power and speed to achieve the optimal result. Under the rockmoving condition, the vehicle controller retrieves the third group of parameter data for parameter adjustment. When adjusting parameters using the third group of data, the third group of data increases power and speed while decreasing the adjustment rate or eliminating the adjustment rate.

[0010] In a further technical solution, under the earthwork condition, the vehicle controller retrieves the second group of parameter data to adjust the parameters, and the adjustment strategy is between the earthwork condition and the rockwork condition.

[0011] A further technical solution involves recognizing the excavator's working image using the image processor, resulting in four actions: the digging stage, the loading stage, the unloading stage, and the return stage.

[0012] In a further technical solution, the actions in the excavation stage are stick excavation plus bucket excavation and partial boom lifting; the actions in the loading stage are boom lifting plus slewing; the actions in the unloading stage are stick unloading plus bucket unloading; and the actions in the return stage are slewing plus boom lowering.

[0013] A further technical solution involves the vehicle controller retrieving Group 1 data from the matching data set for parameter adjustment. When adjusting parameters using Group 1 data, the goal is to reduce fuel consumption, with power and speed adjusted to achieve optimal performance. In the rock excavation condition, the vehicle controller retrieves Group 3 data from the matching data set for parameter adjustment. When adjusting parameters using Group 3 data, the goal is to increase power and speed while decreasing the adjustment rate or eliminating the adjustment rate. In the earthmoving condition, the vehicle controller retrieves Group 2 data from the matching data set for parameter adjustment, with the adjustment strategy falling between that of the earthmoving and rock excavation conditions.

[0014] In a further technical solution, during the excavation phase, the AI ​​camera sends the excavator's status for the next moment to the vehicle controller at fixed intervals, and the vehicle controller adjusts the parameter values ​​to the optimal excavation parameters.

[0015] A further technical solution involves using an AI camera to acquire the slewing angle and boom lifting height, and then sending them to the vehicle controller. Upon receiving the data, the vehicle controller adjusts the proportional parameters of the PNS valve.

[0016] A further technical solution involves the vehicle controller adjusting the proportional parameters of the PNS valve to replenish hydraulic oil and reduce power, thereby ensuring the return position and minimizing oil consumption.

[0017] In a second aspect, the present invention provides an excavator, comprising: a machine vision-based excavator control system as described in any one of the first aspects.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention uses an AI camera to acquire images of the work operation, an image processor to identify the working conditions and actions, and sends this information to the vehicle controller to adjust parameters in advance. This ensures that the digging force is stored up before contacting the load. Compared to the "post-event adjustment" of traditional pump pressure recognition, this reduces action response delay, avoids secondary operations due to insufficient digging power, and improves work efficiency. Furthermore, the image processor recognition avoids the "fuzzy recognition" of pressure sensors, effectively distinguishing similar actions, improving parameter matching rate, and preventing fuel waste or abnormal actions. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is a schematic diagram of the control logic of a machine vision-based overall control system according to the present invention. Detailed Implementation The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0023] Example 1 This embodiment provides a machine vision-based excavator control system. The control system includes an AI camera and a vehicle controller MCU. The AI ​​camera has an embedded image processor, which interacts with the vehicle controller MCU via a CAN bus.

[0024] In this embodiment, two AI cameras are installed at the top two ends of the excavator cab to acquire and store images of the excavator in operation. The two AI cameras complement and verify each other. After acquiring the images, the AI ​​cameras use an image processor to identify the working conditions and actions. In this embodiment, the working conditions obtained after image processor identification include earthwork, earth-rockwork, rockwork, and leveling. The loading and unloading actions obtained after image processor identification include four parts: digging stage, loading stage, unloading stage, and return stage. The digging stage involves boom digging, bucket digging, and partial boom lifting. The loading stage involves boom lifting and slewing. The unloading stage involves boom unloading and bucket unloading. The return stage involves slewing and boom lowering. The difference between loading and unloading is that the boom is raised higher during loading, and the boom unloading is further away during loading.

[0025] In this embodiment, the image processor uses a conventional neural network model to recognize the work image. As long as the goal of action recognition in the same image can be achieved, it will not be elaborated in this embodiment.

[0026] like Figure 1 As shown, the AI ​​camera sends the obtained working conditions and actions to the vehicle controller via the CAN bus, informing the vehicle controller of the current working condition and actions of the excavator. The vehicle controller has pre-stored matching data, and retrieves the matching data based on the received current working conditions and actions to adjust the parameters. The specific parameter adjustment method is as follows: When the excavator is identified as being in earthmoving conditions (light load), the focus is on fuel consumption and driving comfort. The vehicle controller retrieves Group 1 data from the matching data set for parameter adjustment, aiming to reduce fuel consumption by optimizing power and speed. When the excavator is identified as being in rock excavation conditions (heavy load), the focus is on work efficiency. The vehicle controller retrieves Group 3 data from the matching data set for parameter adjustment, increasing power and speed while decreasing or eliminating the speed adjustment rate to ensure digging force. When the excavator is identified as being in earthmoving and rock excavation conditions, with a load between earthmoving and rock excavation, the vehicle controller retrieves Group 2 data from the matching data set for parameter adjustment, with an adjustment strategy between those of earthmoving and rock excavation conditions.

[0027] During the excavation phase, which requires significant power, the AI ​​camera sends the excavator's status to the vehicle controller at fixed intervals when it detects that the bucket is about to enter the soil. The vehicle controller then adjusts the parameters to the optimal excavation parameters. This interval cannot be too long, as it will cause loading delays. In this embodiment, the fixed interval is preferably 10ms. During the loading phase, the AI ​​camera continuously acquires the slewing angle and boom lifting height and sends them to the vehicle controller. Upon receiving these data, the vehicle controller adjusts the proportional parameters of the PNS valve to ensure the matching of slewing and boom movements. During the return phase, as the boom lowers and slewing occurs, the vehicle controller adjusts the proportional parameters of the PNS valve to replenish hydraulic oil for slewing and reduce power to ensure proper return position and minimize fuel consumption.

[0028] In flat terrain, the boom lifts and stick digs, similar to the digging phase. However, flat terrain requires less power and has a lower stick displacement compared to digging; otherwise, fuel waste and nodding issues would occur. The vehicle controller cannot easily distinguish between these two actions. The AI ​​camera, however, can. When it detects flat terrain, it notifies the MCU via ID_X, and the MCU then proactively adjusts the parameters suitable for flat terrain to control the vehicle's hydraulic system.

[0029] Specifically: The valve core of the boom excavator, due to its regenerative function, generally operates relatively quickly and is difficult to control. During boom ascent, the heavy load and high pump pressure result in less flow to the boom. Therefore, the main adjustments are made to the flow distribution and the flow loading process.

[0030] During the leveling phase: The flow rate of the boom main pump is actively reduced, and the flow loading speed of the main pump corresponding to the boom is limited (controlled by current loading speed). Then, the flow distribution and loading speed of the boom main pump are increased. This allows the boom to lift faster in the initial stage of the movement, avoiding the nose-nodding problem in the early stages of leveling. Then, during the process, the boom flow rate is increased and the boom flow rate is reduced to make the movement more coordinated.

[0031] Excavation Phase: Compared to flat ground, this phase requires stronger digging force from the boom and bucket, resulting in a larger displacement and consequently, greater torque. This increases the digging force. Therefore, the displacement of the boom pump is increased. If the total power of the two pumps is 2, the power allocation to the main boom pump can be appropriately set to 1.1-1.2. Even when constant power is achieved, i.e., when excavating heavy objects, the corresponding displacement can still be increased.

[0032] In this embodiment, the vision instrument will adjust and learn over a long period of time. For example, when it becomes unable to dig after 10 or more consecutive diggings, the AI ​​camera will also send an abnormal state ID to the vehicle controller via the CAN bus. After receiving the state ID, the vehicle controller will adjust the parameters linearly. The linear adjustment parameters are selected based on the difference between the pre-calibrated ideal state parameters (maximum and minimum value range) and the current abnormal state parameters, and the adjustment step size is selected, such as incremental PID adjustment.

[0033] Example 2 This embodiment provides an excavator, including: a machine vision-based excavator control system as provided in Embodiment 1.

[0034] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0035] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A machine vision-based excavator control system, characterized in that, include: The AI ​​camera and the vehicle controller interact via a CAN bus; The AI ​​camera is used to acquire images of the excavator in operation. The image processor embedded inside the camera identifies the images of the excavator in operation to obtain the working conditions and actions. The working conditions obtained after the image processor identifies the images of the excavator in operation include earthwork, earth and rock work, rock work, and leveling. Under earthmoving conditions, the vehicle controller retrieves Group 1 data from the parameter data for parameter adjustment. When adjusting parameters using Group 1 data, the goal is to reduce fuel consumption, and the power and speed are adjusted to achieve the optimal values. Under rockmoving conditions, the vehicle controller retrieves Group 3 data from the parameter data for parameter adjustment. When adjusting parameters using Group 3 data, the goal is to increase power and speed while decreasing the adjustment rate or eliminating the adjustment rate. Under the earthwork and rock excavation conditions, the vehicle controller retrieves the second group of parameter data to adjust the parameters, and the adjustment strategy is between that of the earthwork and rock excavation conditions. The system sets parameter data based on operating conditions and actions, and pre-stores the parameter data in the vehicle controller; the AI ​​camera sends the obtained operating conditions and actions to the vehicle controller via the CAN bus in the form of a unique coded ID; the vehicle controller retrieves the corresponding parameter data based on the received unique coded ID and adjusts the parameters accordingly; After the excavator has been digging continuously, the AI ​​camera also sends an abnormal status ID to the vehicle controller via the CAN bus. After receiving the status ID, the vehicle controller adjusts the parameters linearly. The linear adjustment parameters are selected based on the difference between the pre-calibrated ideal status parameters and the current abnormal status parameters, and the adjustment step size is selected accordingly.

2. The excavator control system based on machine vision as described in claim 1, characterized in that, The actions obtained by recognizing the excavator's working image through the image processor include four parts: the digging stage, the loading stage, the unloading stage, and the return stage.

3. The excavator control system based on machine vision as described in claim 2, characterized in that, The actions in the excavation stage are stick excavation, bucket excavation, and partial boom lifting; the actions in the loading stage are boom lifting and slewing; the actions in the unloading stage are stick unloading and bucket unloading; and the actions in the return stage are slewing and boom lowering.

4. The excavator control system based on machine vision as described in claim 2, characterized in that, During the excavation phase, the AI ​​camera sends the excavator's status for the next moment to the vehicle controller at fixed intervals, and the vehicle controller adjusts the parameter values ​​to the optimal excavation parameters.

5. The excavator control system based on machine vision as described in claim 2, characterized in that, During the loading stage, the AI ​​camera acquires the slewing angle and boom lifting height and sends them to the vehicle controller. Upon receiving the data, the vehicle controller adjusts the proportional parameters of the PNS valve.

6. The excavator control system based on machine vision as described in claim 2, characterized in that, During the return phase, the vehicle controller adjusts the proportional parameters of the PNS valve to replenish hydraulic oil and reduce power, in order to ensure the return position and oil consumption.

7. An excavator, characterized in that, include: A machine vision-based excavator control system as described in any one of claims 1-6.

Images