An excavator control method, system, device and readable storage medium

Abstract

The embodiment of the application discloses a kind of excavator control method, system, equipment and readable storage medium, the method includes: identifying target excavation data in device picture selected by user through user equipment, the target excavation data includes excavation point pixel coordinates, unloading point pixel coordinates and target excavation depth;The target excavation data is transmitted to the controller on excavator;The controller converts the excavation point pixel coordinates and unloading point pixel coordinates into corresponding three-dimensional coordinates;The controller controls excavator according to the three-dimensional coordinates of the excavation point and unloading point and target excavation depth under the instruction of user equipment.It realizes the accurate excavation of excavator, reduces the operation difficulty of excavator operator, improves the work efficiency of excavator.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of excavators, and in particular to an excavator control method, system, device and readable storage medium. BACKGROUND

[0002] An excavator is an important engineering vehicle, which can replace manual work for some dangerous, boring or high-intensity work, such as replacing manual work to dig up materials, and the whole process is a cycle, thereby liberating manpower and improving work efficiency.

[0003] The operation learning threshold of an excavator is high, and for repeatable labor, such as ditching, remote automatic excavation function can be used to realize. SUMMARY

[0004] To this end, embodiments of the present application provide an excavator control method, system, device and readable storage medium, which realize accurate excavation of an excavator, reduce the operation difficulty of an excavator operator, and improve the work efficiency of the excavator.

[0005] In order to achieve the above purpose, embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect of embodiments of the present application, an excavator control method is provided, the method comprising:

[0007] identifying target excavation data in a device picture selected by a user through a user device, the target excavation data comprising excavation point pixel coordinates, unloading point pixel coordinates and target excavation depth;

[0008] transmitting the target excavation data to a controller on the excavator;

[0009] the controller converts the excavation point pixel coordinates and the unloading point pixel coordinates into three-dimensional coordinates of the corresponding excavation point and unloading point;

[0010] the controller controls the excavator under the instruction of the user device according to the three-dimensional coordinates of the excavation point and the unloading point and the target excavation depth.

[0011] Optionally, before identifying the target excavation data in the device picture selected by the user device, the method further comprises:

[0012] installing a binocular camera on the excavator to obtain three-dimensional coordinates corresponding to each pixel point in a target range, the three-dimensional coordinates of the pixel point being in a camera coordinate system of the binocular camera;

[0013] The inclination sensor is installed on the swing platform, the boom, the arm, and the bucket of the excavator to obtain the inclination angle of the inclination sensor relative to the ground level, the rotation angular velocity, and the three-dimensional coordinates of the end pose of the bucket of the excavator, wherein the end pose of the bucket of the excavator is taken as the coordinate origin;

[0014] The conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator is calculated.

[0015] The three-dimensional coordinates of the bucket are obtained according to the conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator and the three-dimensional coordinates of the end pose.

[0016] Optionally, the controller controls the excavator according to the three-dimensional coordinates of the digging point and the unloading point and the target digging depth under the instruction of the user device, including:

[0017] The controller controls the end of the bucket to move from the three-dimensional coordinates of the bucket to the three-dimensional coordinates of the digging point, and keeps the tip of the bucket perpendicular to the ground.

[0018] When the target digging depth is reached, the controller controls the tip of the bucket to be horizontal to the ground.

[0019] The controller controls the bucket to rotate to the top of the three-dimensional coordinates of the unloading point and then stops, and opens the bucket to start unloading.

[0020] After unloading, the controller controls the end of the bucket to return to the three-dimensional coordinates of the digging point to continue the above digging steps, and repeats several times until the digging depth reaches the target digging depth. After the unloading of the excavator is completed, the excavation is ended.

[0021] Optionally, the controller converts the digging point pixel coordinates and the unloading point pixel coordinates into the corresponding three-dimensional coordinates of the digging point and the unloading point, including:

[0022] The controller obtains the digging point pixel coordinates based on the binocular camera, and converts the digging point pixel coordinates into the three-dimensional coordinates in the camera coordinate system of the digging point according to the focal length of the binocular camera and the actual depth of the pixel in the picture.

[0023] The controller obtains the unloading point pixel coordinates based on the binocular camera, and converts the unloading point pixel coordinates into the three-dimensional coordinates in the camera coordinate system of the unloading point according to the focal length of the binocular camera and the actual depth of the pixel in the picture.

[0024] According to a second aspect of the embodiments of the present application, a kind of excavator control system is provided, and the system includes,

[0025] The digging data recognition module is used to identify the target digging data selected by the user in the device picture of the user device, and the target digging data includes digging point pixel coordinates, unloading point pixel coordinates and target digging depth.

[0026] a data transmission module configured to transmit the target excavation data to a controller on the excavator;

[0027] a three-dimensional coordinate conversion module configured to convert the excavation point pixel coordinate and the dumping point pixel coordinate into corresponding three-dimensional coordinates of the excavation point and the dumping point by the controller;

[0028] an excavation control module configured to control the excavator under the instruction of the user device according to the three-dimensional coordinates of the excavation point and the dumping point and the target excavation depth by the controller.

[0029] Optionally, the system further comprises:

[0030] a mounting module configured to mount a binocular camera on the excavator to obtain the three-dimensional coordinates of each pixel point in the target range, the three-dimensional coordinates of the pixel point being in a camera coordinate system of the binocular camera, and mount an inclination sensor on a slewing ring, a large arm, a small arm, and a bucket of the excavator to obtain an inclination angle of the inclination sensor relative to a ground plane, a rotation angular velocity, and a three-dimensional coordinate of an end pose of the bucket of the excavator, the three-dimensional coordinate of the end pose being in a coordinate origin of a rotation center of the excavator;

[0031] a calculation module configured to calculate a conversion matrix between the camera coordinate system and the coordinate origin of the rotation center of the excavator, and obtain the three-dimensional coordinates of the bucket according to the conversion matrix between the camera coordinate system and the coordinate origin of the rotation center of the excavator and the three-dimensional coordinate of the end pose.

[0032] Optionally, the excavation control module is configured to:

[0033] control the end of the bucket to move from the three-dimensional coordinates of the bucket to the three-dimensional coordinates of the excavation point, and keep a tip of the bucket perpendicular to the ground;

[0034] when the target excavation depth is reached, control the tip of the bucket to be horizontal to the ground;

[0035] control the bucket to stop after rotating to above the three-dimensional coordinates of the dumping point, open the bucket to start dumping;

[0036] after the dumping, control the end of the bucket to return to the three-dimensional coordinates of the excavation point to continue the above excavation steps, repeat the steps several times until the excavation depth reaches the target excavation depth, and the excavator finishes dumping, and the excavation ends.

[0037] Optionally, the three-dimensional coordinate conversion module is configured to:

[0038] The controller obtains the pixel coordinates of the digging point based on the binocular camera, and converts the pixel coordinates of the digging point into three-dimensional coordinates of the digging point in a camera coordinate system according to the focal length of the binocular camera and the actual depth of the pixel in the picture.

[0039] The controller obtains the pixel coordinates of the unloading point based on the binocular camera, and converts the pixel coordinates of the unloading point into three-dimensional coordinates of the unloading point in a camera coordinate system according to the focal length of the binocular camera and the actual depth of the pixel in the picture.

[0040] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method of the first aspect.

[0041] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer readable instructions executable by a processor to implement the method of the first aspect.

[0042] In summary, the embodiments of the present application provide a control method, system, device and readable storage medium for a excavator, which identifies target digging data in a device picture selected by a user through a user device, the target digging data comprising pixel coordinates of a digging point, pixel coordinates of an unloading point, and a target digging depth; transmits the target digging data to a controller on the excavator; the controller converts the pixel coordinates of the digging point and the pixel coordinates of the unloading point into three-dimensional coordinates of the corresponding digging point and unloading point; and the controller controls the excavator under the instruction of the user device according to the three-dimensional coordinates of the digging point and the unloading point and the target digging depth. The embodiments of the present application realize precise digging of the excavator, reduce the operation difficulty of the operator of the excavator, and improve the working efficiency of the excavator. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0044] The structures, proportions, sizes, etc. shown in the specification are only used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and are not used to limit the conditions that can be implemented by the application, so they do not have technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effect and purpose that can be achieved by the application, should still fall within the scope covered by the disclosed technology.

[0045] Figure 1 A flowchart of a control method of a excavator is provided for the embodiments of the present application.

[0046] Figure 2 A flowchart of a control method of a excavator is provided for the embodiments of the present application.

[0047] Figure 3 A block diagram of a control system of a excavator is provided for the embodiments of the present application.

[0048] Figure 4 A structural diagram of an electronic device is shown for the embodiments of the present application.

[0049] Figure 5 A schematic diagram of a computer-readable storage medium is shown for the embodiments of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described below by specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosed content. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0051] Figure 1 A control method of a excavator is provided for the embodiments of the present application, the method comprising:

[0052] Step 101: identifying target excavation data in a device picture selected by a user through a user device, the target excavation data comprising excavation point pixel coordinates, unloading point pixel coordinates and excavation depth;

[0053] Step 102: transmitting the target excavation data to a controller on the excavator;

[0054] Step 103: the controller converts the excavation point pixel coordinates and unloading point pixel coordinates into three-dimensional coordinates of the corresponding excavation point and unloading point;

[0055] Step 104: the controller controls the excavator under the instruction of the user equipment according to the three-dimensional coordinates of the digging point and the unloading point and the digging depth.

[0056] In a possible implementation, before the target digging data in the device picture selected by the user equipment is identified in step 101, the method further includes:

[0057] A binocular camera is installed on the excavator to obtain the three-dimensional coordinates corresponding to each pixel point in the target range, and the three-dimensional coordinates of the pixel point are in the camera coordinate system of the binocular camera;

[0058] An inclination sensor is installed on the swing platform, the large arm, the small arm, and the bucket of the excavator to obtain the inclination angle of the inclination sensor relative to the ground and the rotation angular velocity, and the three-dimensional coordinates of the end pose of the bucket of the excavator, and the three-dimensional coordinates of the end pose are taken as the coordinate origin of the rotation center of the excavator;

[0059] A conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator is calculated.

[0060] The three-dimensional coordinates of the bucket are obtained according to the conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator and the three-dimensional coordinates of the end pose.

[0061] In a possible implementation, in step 102, the controller converts the pixel coordinates of the digging point and the pixel coordinates of the unloading point into the corresponding three-dimensional coordinates of the digging point and the unloading point, including:

[0062] The controller obtains the pixel coordinates of the digging point based on the binocular camera, and converts the pixel coordinates of the digging point into the three-dimensional coordinates in the camera coordinate system of the digging point according to the focal length of the binocular camera and the actual depth of the pixel in the picture;

[0063] The controller obtains the pixel coordinates of the unloading point based on the binocular camera, and converts the pixel coordinates of the unloading point into the three-dimensional coordinates in the camera coordinate system of the unloading point according to the focal length of the binocular camera and the actual depth of the pixel in the picture.

[0064] In a possible implementation, in step 104, the controller controls the excavator under the instruction of the user equipment according to the three-dimensional coordinates of the digging point and the unloading point and the digging depth, including:

[0065] The controller controls the end of the bucket to move from the three-dimensional coordinates of the bucket to the three-dimensional coordinates of the digging point, and keeps the tip of the bucket perpendicular to the ground; when the target digging depth is reached, the tip of the bucket is controlled to be horizontal to the ground; after the bucket is controlled to rotate to the top of the three-dimensional coordinates of the unloading point and then stop, the bucket is opened to start unloading; after unloading, the end of the bucket is controlled to return to the three-dimensional coordinates of the digging point to continue the above digging steps, and the steps are repeated several times until the digging depth reaches the target digging depth; after the unloading of the excavator is completed, the excavation is completed.

[0066] The excavator control method provided by the embodiment of the application will be described in detail below.

[0067] First, a controller and a control mechanism are installed on the excavator, and all actions of the excavator can be controlled; control signals and sensor signals can be transmitted wirelessly, so that the excavator can be remotely controlled.

[0068] In the embodiment of the application, the positions of an object in two coordinate systems are mainly concerned: a pixel coordinate system: coordinates in a camera picture, which are two-dimensional coordinates, represent the position of the object in a pixel plane; and a camera coordinate system: a coordinate system with the camera as the coordinate center, which is a three-dimensional coordinate, represents the position of the object relative to the camera.

[0069] A binocular camera is installed on the top of the cab of the excavator, and the binocular camera can obtain the three-dimensional coordinates of each pixel point in the field of view. In the pixel coordinate system, the coordinates of a pixel point P in the picture can be represented as:

[0070] P P =(x p ,y p )

[0071] The point in the camera coordinate system is represented as:

[0072] P c =(x c ,y c ,z c )

[0073] Given the focal length f of the camera and the horizontal distance z of the point P in the camera coordinate system, the relationship between the points in the two coordinate systems can be calculated:

[0074] An imu (Inertial Measurement Unit, inclination sensor) is installed on the slewing ring, the large arm, the small arm, and the bucket of the excavator, and the inclination angle and the rotation angular velocity of the imu relative to the ground plane can be obtained; by using forward kinematics, the end pose of the arm of the excavator can be obtained, which is represented as:

[0075] (x imu ,yimu ,z imu )

[0076] The conversion matrix between the camera coordinate system and the excavator rotation center coordinate system can be measured, and is defined as T, which is a 4*4 homogeneous transformation matrix containing rotation and translation relationships; the representation of the excavator bucket end pose in the visual sensor is converted to the representation in the rotation center coordinate system as follows:

[0077]

[0078] The subsequent calculation converts the bucket coordinates into the camera coordinate system by default, and B is used to represent B c =(x' c ,y' c ,z' c )。

[0079] Figure 2 The excavator control process flow diagram is shown. By interacting with the picture transmitted back by the camera installed on the excavator, the excavator can obtain the relative distance from the soil and execute the automatic excavation program to complete the work.

[0080] Step 1: The operator clicks or touches the soil area on the screen of the device; clicking the soil in the picture on the screen corresponds to the soil in the field scene. The device can be a computer or a tablet, etc. The detected soil area is displayed in the remote control interface of the computer or tablet, and all points in the soil area are interactive areas; the operator can mark the point as a digging point by clicking the interactive area in the picture, and a menu pops up after clicking to select the digging depth, and clicking the second point as the unloading point; clicking the same position can cancel the previously selected digging point or unloading point; after the selection is completed, click "start" to start digging, and click "end" to terminate the digging in advance;

[0081] Step 2: The device obtains the corresponding xyz pixel coordinate range of the selected image according to the screen position clicked by the operator, and transmits the pixel coordinate range to the controller on the excavator;

[0082] Step 3: The controller of the excavator converts the pixel coordinate range into the corresponding three-dimensional coordinates through the visual sensor and the depth sensor, that is, the position that the excavator bucket needs to reach and dig; this digging point is recorded as P1=(x1, y1, z1), the digging depth is recorded as D, and the unloading point is recorded as P2=(x2, y2, z2).

[0083] Step 4: The excavator controls the bucket end to move to the specified position and dig in the set way: including the following steps:

[0084] i. Move the bucket end from B to P1=(x1, y1, z1), and keep the bucket tip perpendicular to the ground, which is beneficial to subsequent digging;

[0085] ii. Control the excavator bucket to dig to a certain digging depth, and the excavator bucket tip ends horizontally on the ground;

[0086] iii. Lift the excavator bucket to rotate to the top of P2=(x2, y2, z2) and stop, and open the excavator bucket to unload;

[0087] iv. The excavator returns to the P1 position to continue digging, and repeats several times until the depth reaches D. After the excavation and unloading are completed, this cycle ends.

[0088] The embodiment of the application also provides a control method of an excavator, comprising:

[0089] Step 1: install a binocular camera on the excavator to obtain the three-dimensional coordinates corresponding to each pixel point in the target range, and the three-dimensional coordinates of the pixel point are taken as a camera coordinate system; install an inclination sensor on the swing ring, the large arm, the small arm and the excavator bucket of the excavator to obtain the inclination angle of the inclination sensor relative to the ground level, the rotation angular velocity, and the three-dimensional coordinates of the end pose of the excavator bucket, and the three-dimensional coordinates of the end pose are taken as the coordinate origin of the rotation center of the excavator; calculate the conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator; and obtain the three-dimensional coordinates of the excavator bucket according to the conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator and the three-dimensional coordinates of the end pose.

[0090] Step 2: identify the target excavation data in the device picture selected by the user through the user equipment, and the target excavation data includes the excavation point pixel coordinates, the unloading point pixel coordinates and the excavation depth;

[0091] Step 3: transmit the target excavation data to the controller on the excavator;

[0092] Step 4: the controller obtains the excavation point pixel coordinates based on the binocular camera, converts the excavation point pixel coordinates into the three-dimensional coordinates in the camera coordinate system of the excavation point according to the focal length of the binocular camera and the actual depth of the pixel in the picture; and the controller obtains the unloading point pixel coordinates based on the binocular camera, converts the unloading point pixel coordinates into the three-dimensional coordinates in the camera coordinate system of the unloading point according to the focal length of the binocular camera and the actual depth of the pixel in the picture.

[0093] Step 5: the controller controls the excavation under the instruction of the user equipment according to the three-dimensional coordinates of the excavation point and the unloading point and the excavation depth. Specifically, it can include the following steps:

[0094] The controller controls the end of the bucket to move from the three-dimensional coordinates of the bucket to the three-dimensional coordinates of the digging point, and keeps the tip of the bucket perpendicular to the ground; when the target digging depth is reached, the tip of the bucket is kept horizontal to the ground; the bucket is controlled to rotate to stop above the three-dimensional coordinates of the dumping point, and the bucket is opened to start dumping; after dumping, the end of the bucket is controlled to return to the three-dimensional coordinates of the digging point to continue the above digging steps, and the steps are repeated several times until the target digging depth is reached, and the excavator stops after dumping is completed.

[0095] In summary, the embodiment of the present application provides an excavator control method, which identifies target digging data in a device picture selected by a user through a user device, the target digging data including digging point pixel coordinates, dumping point pixel coordinates and target digging depth; the target digging data is transmitted to a controller on the excavator; the controller converts the digging point pixel coordinates and the dumping point pixel coordinates into corresponding three-dimensional coordinates; and the controller controls digging according to the three-dimensional coordinates of the digging point and the dumping point and the target digging depth under the instruction of the user device. The method realizes precise excavation of the excavator, reduces the operation difficulty of the excavator operator, and improves the working efficiency of the excavator.

[0096] Based on the same technical concept, the embodiment of the present application also provides an excavator control system, as shown in Figure 3 The system comprises:

[0097] A digging data identification module 301 is configured to identify target digging data in a device picture selected by a user through a user device, the target digging data including digging point pixel coordinates, dumping point pixel coordinates and target digging depth;

[0098] A data transmission module 302 is configured to transmit the target digging data to a controller on the excavator;

[0099] A three-dimensional coordinate conversion module 303 is configured to convert the digging point pixel coordinates and the dumping point pixel coordinates into corresponding three-dimensional coordinates of the digging point and the dumping point by the controller;

[0100] A digging control module 304 is configured to control the excavator according to the three-dimensional coordinates of the digging point and the dumping point and the target digging depth by the controller under the instruction of the user device.

[0101] In a possible implementation, the system further comprises:

[0102] The installation module is configured to install a binocular camera on the excavator to obtain a three-dimensional coordinate corresponding to each pixel point in a target range, and the three-dimensional coordinate of the pixel point is in a camera coordinate system of the binocular camera; install an inclination sensor on a swing ring, a large arm, a small arm and a bucket of the excavator to obtain an inclination angle of the inclination sensor relative to a ground plane, a rotation angular velocity and a three-dimensional coordinate of an end pose of the bucket of the excavator, and the three-dimensional coordinate of the end pose is taken as a coordinate origin of a rotation center of the excavator.

[0103] The calculation module is configured to calculate a conversion matrix between the camera coordinate system and a rotation center coordinate system of the excavator, and obtain a three-dimensional coordinate of the bucket according to the conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator and the three-dimensional coordinate of the end pose.

[0104] In a possible implementation, the three-dimensional coordinate conversion module 303 is configured to:

[0105] The controller converts the pixel coordinates of the digging point into three-dimensional coordinates in a camera coordinate system of the digging point according to a focal length of the binocular camera and an actual depth of the pixel in the picture based on the binocular camera obtaining the pixel coordinates of the digging point.

[0106] The controller converts the pixel coordinates of the unloading point into three-dimensional coordinates in a camera coordinate system of the unloading point according to a focal length of the binocular camera and an actual depth of the pixel in the picture based on the binocular camera obtaining the pixel coordinates of the unloading point.

[0107] In a possible implementation, the digging control module 304 is configured to:

[0108] The controller controls the end of the bucket to move from the three-dimensional coordinate of the bucket to the three-dimensional coordinate of the digging point, and keeps the tip of the bucket perpendicular to the ground; when the target digging depth is reached, controls the tip of the bucket to be horizontal to the ground; controls the bucket to stop after rotating to above the three-dimensional coordinate of the unloading point, opens the bucket to start unloading; after unloading, controls the end of the bucket to return to the three-dimensional coordinate of the digging point to continue the above digging step, and repeats the steps several times until the digging depth reaches the target digging depth; after the unloading of the excavator is completed, the digging is ended.

[0109] The embodiment of the present application also provides an electronic device corresponding to the method provided in the foregoing embodiments. Please refer to Figure 4Fig. 20 is a schematic diagram of an electronic device according to some embodiments of the present application. The electronic device 20 can include a processor 200, a memory 201, a bus 202 and a communication interface 203, which are connected through the bus 202; the memory 201 stores a computer program executable on the processor 200, and the processor 200 executes the computer program to perform the method according to any of the preceding embodiments of the present application.

[0110] The memory 201 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 203 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0111] The bus 202 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store a program, and the processor 200 executes the program after receiving an execution instruction. The method disclosed in any of the preceding embodiments of the present application can be applied to the processor 200 or implemented by the processor 200.

[0112] The processor 200 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit or the instruction in the form of software in the processor 200. The processor 200 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201, and combines the hardware to complete the steps of the above method.

[0113] The electronic device provided by the embodiments of the present application and the method provided by the embodiments of the present application have the same beneficial effects as the method they adopt, run or implement.

[0114] The embodiments of the present application also provide a computer readable storage medium corresponding to the method provided by the preceding embodiments. Please refer to Figure 5 The computer readable storage medium shown is an optical disc 30, and a computer program (i.e. program product) is stored on the optical disc 30. When the computer program is run by a processor, the method provided by any of the preceding embodiments is executed.

[0115] It should be noted that examples of the computer readable storage medium can also include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a flash memory or other optical, magnetic storage medium, which will not be described one by one here.

[0116] The computer readable storage medium provided by the above embodiments of the present application and the method provided by the embodiments of the present application have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0117] It is noted that

[0118] The algorithms and displays presented herein are not inherently related to any particular computer, virtual apparatus, or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description above. In addition, the present application is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the application as described herein, and any references below to specific languages are provided for disclosure of enablement only.

[0119] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.

[0120] Similarly, it is to be understood that the mechanical details of the application that have been set forth above in the context of a few illustrative embodiments are for purposes of example only and that various modifications, changes and adaptations will be apparent to those skilled in the art. It is the following claims, including any amendments thereto, that define the scope of the application.

[0121] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into more sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and any method or apparatus so disclosed, can be made unless expressly stated otherwise. Each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features serving the same, equivalent or similar purpose unless expressly stated otherwise.

[0122] Furthermore, those skilled in the art will recognize that references to various embodiments of the application are not intended to limit the scope of the claims that follow this disclosure, but rather can encompass a wide range of embodiments.

[0123] Various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some or all of the components of the virtual machine creation apparatus according to embodiments of the present application. The present application can also be implemented as a program of instructions for performing part or all of the methods described herein, e.g., a computer program and a computer program product. Such program of instructions of the present application can be stored on a computer readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier medium, or in any other form.

[0124] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a conjunction like 'and'. None of the individual devices is thereby obligatory, but one or more of the devices can be mandatory. The use of the word 'at least' followed by a list of one or more members is intended to extend 'at least' to the members of the list. The word 'first' or'second' does not have any order connotation but is used to distinguish between different elements.

[0125] The above description is the preferred specific embodiments of the present application. However, the protection scope of the present application is not limited to this. Any changes or replacements within the technical scope disclosed by the present application can be easily conceived by those skilled in the art. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An excavator control method characterized by, The method comprises: installing a binocular camera on the excavator to obtain three-dimensional coordinates corresponding to each pixel point within a target range, the three-dimensional coordinates of the pixel point being in a camera coordinate system of the binocular camera; installing an inclination sensor on the swing platform, the boom, the arm, and the bucket of the excavator to obtain an inclination angle of the inclination sensor relative to the ground plane, a rotational angular velocity, and a three-dimensional coordinate of an end pose of the bucket of the excavator, the three-dimensional coordinate of the end pose being taken as a coordinate origin of a rotation center of the excavator; calculating a conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator; obtaining a three-dimensional coordinate of the bucket according to the conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator and the three-dimensional coordinate of the end pose; identifying target excavation data in a device picture selected by a user through a user device, the target excavation data comprising pixel coordinates of an excavation point, pixel coordinates of a discharge point, and a target excavation depth; transmitting the target excavation data to a controller on the excavator; converting, by the controller, the pixel coordinates of the excavation point and the pixel coordinates of the discharge point into three-dimensional coordinates of the excavation point and the discharge point; controlling, by the controller, the excavator under an instruction of the user device according to the three-dimensional coordinates of the excavation point and the discharge point and the target excavation depth.

2. The method of claim 1, wherein, The controller controls the excavator under an instruction of the user device according to the three-dimensional coordinates of the excavation point and the discharge point and the target excavation depth, comprising: the controller controls the end of the bucket to move from the three-dimensional coordinate of the bucket to the three-dimensional coordinate of the excavation point, and keeps the tip of the bucket perpendicular to the ground; when the excavation reaches the target excavation depth, the controller controls the tip of the bucket to be horizontal to the ground; after the controller controls the bucket to rotate to above the three-dimensional coordinate of the discharge point and then stops, the controller opens the bucket to start discharging; after the discharge, the controller controls the end of the bucket to return to the three-dimensional coordinate of the excavation point to continue the above excavation step, and repeats the step several times until the excavation depth reaches the target excavation depth; after the discharge of the excavator is completed, the excavation ends.

3. The method of claim 1, wherein, The controller converts the pixel coordinates of the excavation point and the pixel coordinates of the discharge point into three-dimensional coordinates of the excavation point and the discharge point, comprising: the controller obtains the pixel coordinates of the excavation point based on the binocular camera, and converts the pixel coordinates of the excavation point into three-dimensional coordinates in a camera coordinate system of the excavation point according to a focal length of the binocular camera and an actual depth of a pixel in a picture; the controller obtains the pixel coordinates of the discharge point based on the binocular camera, and converts the pixel coordinates of the discharge point into three-dimensional coordinates in a camera coordinate system of the discharge point according to the focal length of the binocular camera and the actual depth of the pixel in the picture.

4. An excavator control system characterized by, The system comprises, a mounting module configured to install a binocular camera on an excavator to obtain three-dimensional coordinates corresponding to each pixel point within a target range, the three-dimensional coordinates of the pixel point being in a camera coordinate system of the binocular camera, and install an inclination sensor on a swing platform, a boom, an arm, and a bucket of the excavator to obtain an inclination angle of the inclination sensor relative to a ground plane, a rotational angular velocity, and a three-dimensional coordinate of an end pose of the bucket of the excavator, the three-dimensional coordinate of the end pose being taken as a coordinate origin of a rotation center of the excavator; The computing module is configured to calculate a conversion matrix between a camera coordinate system and a rotation center coordinate system of the excavator, and to obtain three-dimensional coordinates of the bucket based on the conversion matrix between the camera coordinate system and the rotation center coordinate system of the excavator and the three-dimensional coordinates of the end position. The excavating data identifying module is configured to identify target excavating data in a device image selected by the user through the user device, the target excavating data including a pixel coordinate of a digging point, a pixel coordinate of a dumping point, and a target excavating depth. The data transmission module is configured to transmit the target excavating data to a controller on the excavator. The three-dimensional coordinate conversion module is configured to convert the pixel coordinates of the digging point and the dumping point into three-dimensional coordinates of the digging point and the dumping point, respectively, by the controller. The excavating control module is configured to control the excavator based on the three-dimensional coordinates of the digging point and the dumping point and the target excavating depth by the controller under the instruction of the user device.

5. The system of claim 4, wherein, The excavating control module is configured to: control the end of the bucket to move to the three-dimensional coordinates of the digging point from the three-dimensional coordinates of the bucket, and keep the tip of the bucket perpendicular to the ground; control the tip of the bucket to be horizontal to the ground when the target excavating depth is reached; control the bucket to stop rotating above the three-dimensional coordinates of the dumping point, open the bucket, and start dumping; control the end of the bucket to return to the three-dimensional coordinates of the digging point after dumping to continue the above-mentioned excavating steps, and repeat the steps until the excavating depth reaches the target excavating depth, and the excavator finishes dumping, and the excavating ends.

6. The system of claim 4, wherein, The three-dimensional coordinate conversion module is configured to: convert the pixel coordinate of the digging point into three-dimensional coordinates of the digging point in the camera coordinate system based on the focal length of the binocular camera and the actual depth of the pixel in the image by the controller; convert the pixel coordinate of the dumping point into three-dimensional coordinates of the dumping point in the camera coordinate system based on the focal length of the binocular camera and the actual depth of the pixel in the image by the controller.

7. An electronic device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, A computer readable instruction is stored on the memory, and the computer readable instruction is executable on the processor to implement the method of any one of claims 1-3.

Images