Excavator control method, device, equipment and storage medium

Abstract

The present application provides an excavator control method, apparatus, device, and storage medium. The method includes obtaining an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled. The method then determines a target excavation position of the excavator to be controlled from the working environment based on screen pixel coordinates input for the image of the working environment and the point cloud data of the working environment. The method then controls the excavator to be controlled to perform excavation operations at the target excavation position based on the current state parameters. Determining the excavation position based on screen pixel coordinates input for the image of the working environment and the point cloud data of the working environment improves the accuracy of the excavation position, enabling automated one-click excavation operations, effectively reducing operational difficulty and improving excavator control accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an excavator control method, device, equipment and storage medium. Background Art

[0002] Construction machinery is an important component of the equipment industry and is divided into excavators, earth-moving and transporting machinery, construction lifting machinery, industrial vehicles, etc. Among them, excavators use buckets to dig materials above or below the surface and load them into transport vehicles or unload them into stockpiles.

[0003] At present, most ways to remotely control excavators are to simulate the cockpit. By simulating the cockpit of a real excavator and integrating video transmission, the excavator can be remotely controlled. Remote control of the excavator not only allows the operator to work in a comfortable environment, but also ensures the operator's personal safety and improves the quality and efficiency of the operation.

[0004] However, remote control of the excavator through remote video transmission can easily cause the operator to misjudge the excavation position and depth, which may result in empty excavation, excavator rollover, etc. Summary of the Invention

[0005] In view of this, embodiments of the present application provide an excavator control method, apparatus, device, and storage medium to accurately determine the excavation position and improve the excavator control accuracy.

[0006] In a first aspect, an embodiment of the present application provides an excavator control method, comprising:

[0007] Acquire an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled;

[0008] determining a target excavation position of the excavator to be controlled from the working environment based on screen pixel coordinates of an image input of the working environment and point cloud data of the working environment;

[0009] According to the current state parameter, the excavator to be controlled is controlled to perform excavation work on the target excavation position.

[0010] In a second aspect, an embodiment of the present application further provides an excavator control device, comprising:

[0011] an acquisition module, configured to acquire an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled;

[0012] a determination module for determining a target excavation position of the excavator to be controlled from the working environment based on screen pixel coordinates of an image input of the working environment and point cloud data of the working environment;

[0013] A control module is used to control the excavator to be controlled to perform an excavation operation on the target excavation position according to the current state parameter.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to execute the excavator control method described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the excavator control method described in the first aspect is executed.

[0016] The present application provides an excavator control method, apparatus, device, and storage medium. The method includes obtaining an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled. The method then determines a target excavation position of the excavator to be controlled from the working environment based on screen pixel coordinates input for the image of the working environment and the point cloud data of the working environment. The method then controls the excavator to be controlled to perform excavation operations at the target excavation position based on the current state parameters. Determining the excavation position based on screen pixel coordinates input for the image of the working environment and the point cloud data of the working environment improves the accuracy of the excavation position, enabling automated one-click excavation operations, effectively reducing operational difficulty and improving excavator control accuracy.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0019] Figure 1A schematic diagram of the camera coordinate system and image coordinate system provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the image coordinate system and pixel coordinate system provided in an embodiment of the present application;

[0021] Figure 3 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 1 ;

[0022] Figure 4 A schematic diagram of an excavator to be controlled provided in an embodiment of the present application;

[0023] Figure 5 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 2 ;

[0024] Figure 6 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 3 ;

[0025] Figure 7 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 4 ;

[0026] Figure 8 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 5 ;

[0027] Figure 9 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 6 ;

[0028] Figure 10 A schematic diagram of a display interface is provided for an embodiment of the present application;

[0029] Figure 11 A schematic structural diagram of an excavator control device provided in an embodiment of the present application;

[0030] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0032] For the current solution of remotely controlling an excavator through a simulated cockpit, for tasks such as excavation or dumping that require multiple repetitive operations, the operator still needs to continuously perform complex operations in the simulated cockpit. The operator needs to observe with the naked eye through one or more planar remote videos, manually judge the location and depth of excavation, and use a handle similar to that in the excavator cockpit in the simulated cockpit for remote control.

[0033] However, the above solution has the following defects:

[0034] (1) Existing remote control requires operation in a simulated cockpit, which places extremely high demands on the establishment of a remote control environment. During the remote control process, each excavation action requires manual adjustment of the excavator's arm position and bucket angle.

[0035] (2) When using a remote control joystick to perform operations that require complex movements, such as digging, the operator must be highly focused due to the complexity of the operation and the delay caused by data transmission. Therefore, the efficiency and quality of the construction work often vary depending on the level of the operator.

[0036] (3) Flat remote video transmission is not like the on-site operation that allows the operator to experience the surrounding environment firsthand, which can easily lead to the operator misjudging the excavation position and depth during remote control. During the excavation operation, empty excavation may occur, and in severe cases, the excavator may even overturn.

[0037] From this, we can see that currently remote excavators require human judgment and repeated complex operations to complete excavation operations in real environments. Although this remote control method improves the operator's working environment, it does not effectively release the operator's labor and lower the operating threshold.

[0038] Based on this, the present application provides an excavator control method, which provides a simple semi-automatic excavation method for current excavators, allowing the operator to autonomously control and complete the excavation operation by only inputting the screen pixel coordinates of the image of the working environment (such as determined by clicking the mouse). That is, it is of practical significance to integrate the semi-automatic excavation function into the image transmission interface of the remote control excavator. The operator only needs to click on the target position to be excavated on the remote interface, and the excavator can automatically excavate the target pixel area, realizing the automated operation of digging wherever the button is clicked, effectively reducing the difficulty of operation and improving work efficiency.

[0039] Before introducing the technical solution of this application, the four coordinate systems involved in this application are first explained:

[0040] World coordinate system: the absolute coordinate system of the system.

[0041] Camera coordinate system: The coordinate system corresponding to the lidar, usually with the lidar as the coordinate origin.

[0042] Image coordinate system: The default coordinate system in the camera processing logic, that is, the coordinate system corresponding to the image of the working environment, usually with the center of the image as the coordinate origin.

[0043] Pixel coordinate system: The coordinate system used in normal visual processing, that is, the coordinate system corresponding to screen pixels, usually with the upper left corner of the image as the coordinate origin.

[0044] (1) The transformation relationship from the world coordinate system to the camera coordinate system is as follows:

[0045]

[0046] (x c ,y c , z c ) and (x w ,y w , z w ) are the positions of the point in the camera coordinate system and the world coordinate system respectively. There is no projection transformation here, only the reference coordinate system is changed. The R matrix refers to the rotation matrix from the world coordinate system to the camera coordinate system, and T is the translation variable.

[0047] (2) Conversion from camera coordinate system to image coordinate system:

[0048] Figure 1 A schematic diagram of the camera coordinate system and image coordinate system provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, after projection transformation, the image coordinate system is a two-dimensional coordinate system xy, the coordinate origin is o, there is no Z axis component, the coordinates of point p are (x, y); the camera coordinate system is x c y c zc , the coordinate origin is o c , the coordinates of point P are (X c , Y c , Z c ).

[0049] The imaging plane of the camera is at a distance o from the optical center c The distance is the focal length f. Therefore, when converting from the camera coordinate system to the image coordinate system, a scale factor Z is introduced c , this scale factor is related to f.

[0050] The camera coordinate system to the image coordinate system satisfies the pinhole imaging model, which can be obtained by the simple principle of similar triangles:

[0051] ΔABO c ~ΔoBO c

[0052] ΔPBO c ~ΔpBO c

[0053] The transformation relationship from the camera coordinate system to the image coordinate system is:

[0054]

[0055] (3) Conversion from image coordinate system to pixel coordinate system:

[0056] Figure 2 A schematic diagram of the image coordinate system and pixel coordinate system provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, the image coordinate system is xy, and the coordinate origin is o(u c , v c ), the pixel coordinate system is uv, and the origin coordinate is o uv , the coordinate origin in the image coordinate system is in the middle of the image, the coordinate origin in the pixel coordinate system is in the upper left corner of the image, the measurement unit of the image coordinate system is mm, the measurement unit of the pixel coordinate system is pixel, the coordinates of point p are (x, y), then:

[0057]

[0058] That is, the transformation relationship from the image coordinate system to the pixel coordinate system is:

[0059]

[0060] dx and dy represent the width and height of each pixel in the pixel coordinate system.

[0061] The excavator control method provided in this application is described below with reference to several specific embodiments.

[0062] Figure 3 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 1 The execution subject of this embodiment can be an electronic device, such as a terminal device, a server, or other device with a display interface.

[0063] like Figure 3 As shown, the method may include:

[0064] S101 , obtaining an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled.

[0065] Among them, the camera and the laser radar can be set on the excavator to be controlled, the camera is used to collect images of the working environment of the excavator to be controlled, and the laser radar is used to collect point cloud data of the working environment of the excavator to be controlled.

[0066] The current state parameters of the excavator to be controlled may include: angle and angular velocity. The excavator to be controlled may be equipped with an inertial measurement unit (IMU) for collecting the angle and angular velocity.

[0067] The camera can send the collected image of the working environment to the electronic device, the lidar can send the collected point cloud data of the working environment to the electronic device, and the IMU can send the angle and angular velocity of the corresponding component to the electronic device, so as to obtain the image of the working environment of the excavator to be controlled, the point cloud data of the working environment, and the current state parameters of the excavator to be controlled.

[0068] In some embodiments, the current state parameters of the excavator to be controlled include the current state parameters of multiple joints on the excavator to be controlled. The current state of multiple components on the excavator to be controlled may include, for example, an upper arm, an lower arm, a bucket, and a cockpit. Inertial measurement units (IMUs) may be installed on the joints between the components to collect the angles and angular velocities of the joints.

[0069] Figure 4 A schematic diagram of an excavator to be controlled provided in an embodiment of the present application is shown in FIG. Figure 4As shown, the base_link of the excavator's overall coordinate system (i.e., the world coordinate system), the upper_body_link of the upper body coordinate system (i.e., the cockpit coordinate system), the boom_link of the upper arm coordinate system, the arm_link of the forearm coordinate system, and the gripper_link of the bucket coordinate system are defined. Among them, the coordinate origin of the cockpit coordinate system is at the center of the joint between the cockpit and the track (i.e., the cockpit joint), the coordinate origin of the boom_link of the upper arm coordinate system is at the center of the joint between the boom and the cockpit (i.e., the boom joint), the coordinate origin of the forearm coordinate system is at the center of the joint between the forearm and the upper arm (i.e., the arm joint), and the coordinate origin of the bucket coordinate system is at the center of the joint between the bucket and the forearm (i.e., the bucket joint).

[0070] Among them, there is a fixed conversion relationship between the coordinate systems of adjacent components. An IMU is set at the bucket joint, an IMU is set at the forearm joint, an IMU is set at the boom joint, and an IMU is set at the cabin joint. The IMU of the bucket joint is used to collect the angle and angular velocity of the joint between the forearm and the bucket in the bucket coordinate system, the IMU of the forearm joint is used to collect the angle and angular velocity of the forearm joint in the forearm coordinate system, the IMU of the boom joint is used to collect the angle and angular velocity of the boom joint in the boom coordinate system, and the IMU of the cabin joint is used to collect the angle and angular velocity of the cabin joint in the cabin coordinate system.

[0071] Then, according to the conversion relationship between the bucket coordinate system and the world coordinate system, the conversion relationship between the forearm coordinate system and the world coordinate system, the conversion relationship between the arm coordinate system and the world coordinate system, and the conversion relationship between the cockpit coordinate system and the world coordinate system, the above-collected joint angles and angular velocities are converted to the world coordinate system.

[0072] It should be noted that during the movement of the excavator, the boom coordinate system boom_link, the arm coordinate system arm_link, and the bucket coordinate system gripper_link rotate around the y-axis. The downward movement of the boom, arm, and bucket is a counterclockwise rotation around the y-axis, and the joint angles of the boom, arm, and bucket gradually increase; the upward movement of the boom, arm, and bucket is a clockwise rotation around the y-axis, and the joint angles of the boom, arm, and bucket gradually decrease.

[0073] The cockpit coordinate system upper_body_link rotates around the z-axis. When the cockpit moves to the left, it rotates counterclockwise around the z-axis, and the joint angle of the cockpit gradually increases. When the cockpit moves to the right, it rotates clockwise around the z-axis, and the joint angle of the cockpit gradually decreases.

[0074] In addition, the joint zero position (i.e., the joint angle is 0) is defined as the angle between the world coordinate system boom_link and the arm coordinate system arm_base_link is 1 radian, the forearm coordinate system arm_link and the world coordinate system boom_link are 1.5 radians, and the bucket coordinate system gripper_link is parallel to the forearm coordinate system arm_link. For example, if the forearm rotates upward by 1 radian based on this posture, the output joint angle is -1 radian.

[0075] S102 : Determine a target excavation position of the excavator to be controlled in the working environment according to the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment.

[0076] The electronic device displays an image of the working environment, and screen pixel coordinates can be input for the image of the working environment. The screen pixel coordinates are the pixel coordinates of the target excavation position in the screen pixel coordinate system. By inputting the screen pixel coordinates of the target excavation position and combining them with the point cloud data of the working environment, the point cloud data corresponding to the screen pixel coordinates can be determined from the point cloud data of the working environment, and the target excavation position of the excavator to be controlled in the working environment can be determined based on the point cloud data corresponding to the screen pixel coordinates.

[0077] In some embodiments, the screen pixel coordinates of the image input of the working environment can be: the coordinates of the screen pixels corresponding to the click operation of the image input of the working environment. The click operation can be achieved by touch or mouse click. That is to say, by performing a click operation on the image input of the working environment, the screen pixel coordinates corresponding to the click operation can be determined, and then combined with the point cloud data of the working environment, the target excavation position of the excavator to be controlled in the working environment can be determined.

[0078] S103 : Control the excavator to be controlled to perform excavation work at the target excavation position according to the current state parameters.

[0079] According to the current state parameters of the excavator to be controlled and the target excavation position, the motion trajectory of the excavator to be controlled can be planned, and according to the motion trajectory, the excavator to be controlled can be controlled to perform excavation operations at the target excavation position.

[0080] In some embodiments, the current state parameters include: current state parameters of multiple components on the excavator to be controlled; and controlling the excavator to be controlled to perform an excavation operation at a target excavation location based on the current state parameters includes:

[0081] A trajectory planner is used to generate a motion trajectory of each component according to the current state parameters of each component and the target excavation position; and each component is controlled to move according to the motion trajectory of each component so that the excavator to be controlled performs excavation operations at the target excavation position.

[0082] After determining the target excavation position from the working environment, a trajectory planner can be used, combined with kinematic equations, to generate the motion trajectory of each component based on the current state parameters of each component and the target excavation position. Then, based on the motion trajectory of each component, a speed controller is used to output a corresponding control signal, and each component is controlled to move based on the control signal, so that the excavator to be controlled performs excavation operations at the target excavation position, wherein the control signal can be a pulse width modulation (PWM) signal.

[0083] It should be noted that the point cloud data corresponding to the screen pixel coordinates can be converted from the camera coordinate system corresponding to the lidar to the world coordinate system to obtain the three-dimensional coordinates of the target excavation position, and then the motion trajectory of each component can be generated based on the current state parameters of each component and the three-dimensional coordinates of the target excavation position.

[0084] In the excavator control method of this embodiment, a semi-automatic excavation method is provided for the excavator. The operator only needs to input the screen pixel coordinates of the image of the working environment to excavate the target excavation position, realizing the automated operation of digging wherever the key is clicked, effectively reducing the operating difficulty and improving the working effect. At the same time, it also improves the control accuracy of the excavator. For excavation actions that require simultaneous control of the large and small arms, cockpit, and bucket rotation, it can be completed by simply clicking the mouse on the client interface, effectively lowering the technical threshold of the excavator operator and improving work efficiency.

[0085] exist Figure 3 Based on the embodiment, the following Figure 5 The embodiment describes a possible implementation process of the target excavation position.

[0086] Figure 5 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 2 ,like Figure 4 As shown, before determining the target excavation position of the excavator to be controlled from the working environment based on the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment, the method may further include:

[0087] S201 : Convert the image of the working environment from the image coordinate system to the pixel coordinate system to obtain pixel coordinates corresponding to the image of the working environment.

[0088] According to the conversion relationship between the image coordinate system and the pixel coordinate system, the image of the working environment is converted from the image coordinate system to the pixel coordinate system to obtain the pixel coordinates corresponding to the image of the working environment.

[0089] S202 : jointly calibrate the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment to obtain calibration data.

[0090] In order to achieve spatial and temporal synchronization of the data collected by the lidar and camera, the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment can be jointly calibrated to obtain calibration data. The calibration data is used to indicate the correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment, that is, to establish a correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment.

[0091] Accordingly, according to the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment, a target excavation position of the excavator to be controlled is determined from the working environment, including:

[0092] S203 : Determine the point cloud data corresponding to the screen pixel coordinates from the point cloud data of the working environment according to the corresponding relationship.

[0093] Among them, the pixel coordinates corresponding to the image of the working environment include: screen pixel coordinates. According to the correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment, the point cloud data corresponding to the screen pixel coordinates can be determined from the point cloud data of the working environment.

[0094] S204: Determine the target excavation position from the working environment based on the point cloud data corresponding to the screen pixel coordinates.

[0095] Among them, the target excavation position can be a position in the world coordinate system, and the point cloud data corresponding to the screen pixel coordinates can be data in the camera coordinate system. In order to determine the target excavation position based on the input screen pixel coordinates, the target excavation position in the working environment can be determined based on the conversion relationship between the camera coordinate system and the world coordinate system and the point cloud data corresponding to the screen pixel coordinates.

[0096] exist Figure 5 Based on the embodiment, the following Figure 6 、 Figure 7 The embodiment describes two possible implementations of the target excavation position.

[0097] Figure 6 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 3 ,like Figure 6 As shown, based on the point cloud data corresponding to the screen pixel coordinates, determining the target excavation position from the working environment may include:

[0098] S301. Convert the point cloud data corresponding to the screen pixel coordinates from the camera coordinate system corresponding to the laser radar to the world coordinate system to obtain the three-dimensional coordinates corresponding to the screen pixel coordinates.

[0099] S302: Determine the target excavation position in the working environment according to the three-dimensional coordinates corresponding to the screen pixel coordinates.

[0100] According to the conversion relationship between the pixel coordinate system and the camera coordinate system, the point cloud data corresponding to the screen pixel coordinates is converted from the camera coordinate system corresponding to the lidar to the world coordinate system to obtain the three-dimensional coordinates corresponding to the screen pixel coordinates. Then, based on the three-dimensional coordinates corresponding to the screen pixel coordinates, the position corresponding to the three-dimensional coordinates is determined from the working environment as the target excavation position.

[0101] Figure 7 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 4 ,like Figure 7 As shown, based on the point cloud data corresponding to the screen pixel coordinates, the target excavation position is determined from the working environment, including:

[0102] S401 , performing pixel expansion on the point cloud data of the working environment according to the point cloud data corresponding to the screen pixel coordinates to obtain target point cloud data within a preset pixel range.

[0103] In order to avoid errors caused by clicking on the interface and improve the accuracy of the excavation position, pixels can be expanded in all directions in the point cloud data of the working environment with the point cloud data corresponding to the screen pixel coordinates as the center. For example, 10 pixels can be expanded to obtain the target point cloud data within the preset pixel range. The target point cloud data can be a pixel range with the point cloud data corresponding to the screen pixel coordinates as the center and expanded by 10 pixels.

[0104] S402: Determine the target excavation position from the working environment based on the target point cloud data within a preset pixel range.

[0105] Based on the conversion relationship between the camera coordinate system and the world coordinate system, the target excavation position in the working environment can be determined according to the target point cloud data within the preset pixel range. If the target point cloud data within the preset pixel range is the point cloud data corresponding to the screen pixel coordinates, the point cloud data corresponding to the screen pixel coordinates can be converted from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the screen pixel coordinates, and based on the three-dimensional coordinates, the position corresponding to the three-dimensional coordinates in the working environment is determined as the target excavation position.

[0106] In an optional embodiment, determining a target excavation location from an operating environment based on target point cloud data within a preset pixel range includes:

[0107] The target point cloud data is converted from the camera coordinate system corresponding to the lidar to the pixel coordinate system to obtain the pixel coordinates corresponding to the target point cloud data; based on the pixel coordinates corresponding to the target point cloud data, the target excavation position is determined from the working environment.

[0108] According to the conversion relationship between the camera coordinate system and the image coordinate system, the target point cloud data can be converted from the camera coordinate system corresponding to the lidar to the image coordinate system to obtain the image coordinates corresponding to the target point cloud data. The image coordinates are then converted from the image coordinate system to the pixel coordinate system to obtain the pixel coordinates corresponding to the target point cloud data. Based on the pixel coordinates corresponding to the target point cloud data, the position corresponding to the pixel coordinates corresponding to the target point cloud data is determined from the working environment as the target excavation position.

[0109] In an optional embodiment, determining the target excavation position from the working environment based on the pixel coordinates corresponding to the target point cloud data includes:

[0110] Cluster the pixel coordinates corresponding to the target point cloud data to obtain multiple pixel coordinate categories; calculate the degree of fit between the multiple pixel coordinate categories and the screen pixel coordinates; determine the target pixel coordinate category from the multiple pixel coordinate categories based on the degree of fit; and determine the target excavation position from the operating environment based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category.

[0111] Among them, since the target point cloud data will show the characteristics of multiple clusters combined, it is not conducive to the separate identification of multiple targets. Therefore, it is necessary to separate the single target cluster, that is, cluster the pixel coordinates corresponding to the target point cloud data to obtain multiple pixel coordinate categories, and calculate the degree of fit between the multiple pixel coordinate categories and the screen pixel coordinates respectively. Then, based on the degree of fit, the target pixel coordinate category of the preset condition is determined from the multiple pixel coordinate categories. The preset condition can be the one with the highest degree of fit.

[0112] Then, according to the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, a position corresponding to the point cloud data is determined as a target excavation position from the working environment, wherein the number of pixel coordinates corresponding to the target point cloud data is multiple.

[0113] It should be noted that the pixel coordinates corresponding to the target point cloud data can be clustered based on the kd-tree principle to establish a topological relationship between points for fast point search, and then the target pixel coordinate category with the highest consistency with the screen pixel coordinates is searched out, wherein the degree of consistency between the pixel coordinate category and the screen pixel coordinates can be calculated as follows: the mean coordinates of the pixel coordinates in the pixel coordinate category are calculated, and the degree of consistency between the mean coordinates and the screen pixel coordinates is used as the degree of consistency between the pixel coordinate category and the screen pixel coordinates, wherein the degree of consistency can be similarity, which can be calculated using the distance between pixel coordinates. The larger the distance, the more dissimilar, and the smaller the distance, the more similar.

[0114] In an optional embodiment, determining the target excavation position from the working environment based on the pixel coordinates corresponding to the target point cloud data includes:

[0115] The degree of agreement between a plurality of pixel coordinates corresponding to the target point cloud data and the screen pixel coordinates is calculated respectively; based on the degree of agreement; the target pixel coordinates are determined from the plurality of pixel coordinates; and the target excavation position is determined from the working environment based on the target pixel coordinates.

[0116] Among them, the number of pixel coordinates corresponding to the target point cloud data is multiple, and the degree of fit between the multiple pixel coordinates corresponding to the target point cloud data and the screen pixel coordinates can be calculated separately, and the pixel coordinates that meet the preset conditions are determined from the multiple pixel coordinates as the target pixel coordinates, and the preset condition can be the highest degree of fit.

[0117] Then, based on the correspondence between the pixel coordinates of the work environment image and the point cloud data of the work environment, the point cloud data corresponding to the target pixel coordinates is obtained. The point cloud data corresponding to the target pixel coordinates is converted from the camera coordinate system to the world coordinate system to obtain the 3D coordinates corresponding to the target pixel coordinates. The position corresponding to these 3D coordinates in the work environment is determined as the target excavation location. The target excavation location is determined by the degree of agreement, which improves the accuracy of the target excavation location.

[0118] exist Figure 7 Based on the embodiment, the following Figure 8 、 Figure 9 The embodiment describes two possible implementations of the target excavation position.

[0119] Figure 8 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 5 ,like Figure 8 As shown, based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, the target excavation position is determined from the working environment, including:

[0120] S501, converting the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category;

[0121] S502: Determine a target excavation position in the operating environment according to the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category.

[0122] If the number of pixel coordinates in the target pixel coordinate category is one, that is, the pixel coordinates in the target pixel coordinate category have the highest degree of agreement with the screen pixel coordinates, the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category can be converted from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category. The location corresponding to the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category in the operating environment is then determined as the target excavation location. Determining the target excavation location based on the degree of agreement improves the accuracy of the target excavation location.

[0123] Figure 9 Schematic diagram of the process of the excavator control method provided in the embodiment of the present application Figure 6 ,like Figure 9 As shown, based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, the target excavation position is determined from the working environment, including:

[0124] S601: Calculate the degree of agreement between a plurality of pixel coordinates in a target pixel coordinate category and the screen pixel coordinates.

[0125] S602: Determine target pixel coordinates from multiple pixel coordinates based on the degree of fit.

[0126] There are multiple pixel coordinates in the target pixel coordinate category.

[0127] The degrees of agreement between the multiple pixel coordinates in the target pixel coordinate category and the screen pixel coordinates are calculated respectively, and based on the degrees of agreement, the pixel coordinates that meet the preset conditions are determined as the target pixel coordinates from the multiple pixel coordinates. The preset conditions may be, for example, the highest degree of agreement.

[0128] S603: Determine the target excavation position from the working environment based on the point cloud data corresponding to the target pixel coordinates.

[0129] According to the correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment, the point cloud data corresponding to the target pixel coordinates is determined from the target point cloud data. The point cloud data of the working environment includes: target point cloud data.

[0130] The point cloud data corresponding to the target pixel coordinates is converted from the camera coordinate system to the world coordinate system to obtain the 3D coordinates corresponding to the target pixel coordinates. The location corresponding to the 3D coordinates of the target pixel coordinates in the working environment is then determined as the target excavation location. The target excavation location is determined by the degree of fit, which improves the accuracy of the target excavation location.

[0131] In an optional embodiment, before determining the target excavation position from the working environment based on the point cloud data corresponding to the target pixel coordinates, the method may further include:

[0132] Determine the target pixel area on the display screen according to the calibration data; and determine whether the target pixel coordinates are within the target pixel area.

[0133] The pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment are jointly calibrated to obtain calibration data, which is used to indicate the correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment.

[0134] Among them, there may be some pixel coordinates in the calibration data that have no corresponding relationship, that is, there is no point cloud data corresponding to this part of the pixel coordinates, that is, the calibration data is used to indicate the correspondence between the pixel coordinates of the target pixel area on the display interface and the point cloud data, that is, according to the calibration data, the target pixel area on the display screen can be determined, and the target pixel area corresponds to the point cloud data.

[0135] After determining the target pixel coordinates, it may be determined whether the target pixel coordinates are located within the target pixel area.

[0136] Accordingly, the target excavation location is determined from the working environment based on the point cloud data corresponding to the target pixel coordinates, including:

[0137] If the target pixel coordinates are within the target pixel area, the point cloud data corresponding to the target pixel coordinates are converted from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the target pixel coordinates; according to the three-dimensional coordinates corresponding to the target pixel coordinates, the target excavation position in the working environment is obtained.

[0138] If the target pixel coordinates are within the target pixel area, it means that the corresponding point cloud data can be determined based on the target pixel coordinates, and then the target excavation position can be determined based on the corresponding point cloud data. The point cloud data corresponding to the target pixel coordinates is converted from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the target pixel coordinates. The position corresponding to the three-dimensional coordinates corresponding to the target pixel coordinates is determined from the working environment as the target excavation position.

[0139] If the target pixel coordinates are not within the target pixel area, it means that the corresponding point cloud data cannot be determined based on the target pixel coordinates, and thus the target excavation position cannot be determined.

[0140] In an optional embodiment, before determining the target excavation position of the excavator to be controlled from the working environment based on the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment, the method may further include:

[0141] The target pixel area on the display screen is determined according to the calibration data, and the target pixel area corresponds to point cloud data.

[0142] The screen pixel coordinates input for the image of the working environment include: screen pixel coordinates input for the target pixel area.

[0143] In this semi-automatic mining function, the working range is generated by calibrating the point cloud data and image data through the calibrated lidar and monocular camera, and activating this part of the range as an optional working range.

[0144] Figure 10 A schematic diagram of a display interface is provided for an embodiment of the present application, such as Figure 10 As shown, the display interface shows: an image of the working environment, a target pixel area determined according to the calibration data as the area in the black bold box, that is, the area in the black bold box is the selectable mining range, and screen pixel coordinates input for the image of the working environment, including: screen pixel coordinates input for the target pixel area, that is, inputting screen pixel coordinates for the target pixel area is valid, such as clicking in this area, while inputting screen pixel coordinates in other areas is invalid.

[0145] It should be noted that in actual applications, the electronic device can be installed with an excavator client. After logging into the excavator client, select the "click to dig" function to activate the selectable excavation range (i.e., the target pixel area). Click within the target pixel area, and the electronic device can obtain the screen pixel coordinates corresponding to the click operation, and then obtain the target excavation position of the excavator to be controlled, provide target parameters for the automatic control of the excavator, and the excavator will automatically complete an excavation operation.

[0146] Based on the same inventive concept, an excavator control device corresponding to the excavator control method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned excavator control method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0147] Figure 11 This is a schematic diagram of the structure of the excavator control device provided in the embodiment of the present application. The device can be used in electronic equipment, such as Figure 11 As shown, the device may include:

[0148] An acquisition module 701 is configured to acquire an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled;

[0149] A determination module 702 is configured to determine a target excavation position of the excavator to be controlled from the working environment based on the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment;

[0150] The control module 703 is used to control the excavator to be controlled to perform excavation work at the target excavation position according to the current state parameters.

[0151] In an optional embodiment, the device further comprises:

[0152] A conversion module 704 is used to convert the image of the working environment from the image coordinate system to the pixel coordinate system to obtain pixel coordinates corresponding to the image of the working environment;

[0153] A calibration module 705 is configured to jointly calibrate the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment to obtain calibration data, wherein the calibration data is used to indicate the correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment;

[0154] The determination module 702 is specifically configured to:

[0155] According to the corresponding relationship, the point cloud data corresponding to the screen pixel coordinates is determined from the point cloud data of the working environment;

[0156] Based on the point cloud data corresponding to the screen pixel coordinates, the target excavation location is determined from the working environment.

[0157] In an optional implementation, the determination module 702 is specifically configured to:

[0158] Convert the point cloud data corresponding to the screen pixel coordinates from the camera coordinate system corresponding to the lidar to the world coordinate system to obtain the three-dimensional coordinates corresponding to the screen pixel coordinates;

[0159] Determine the target excavation position in the working environment based on the three-dimensional coordinates corresponding to the screen pixel coordinates.

[0160] In an optional implementation, the determination module 702 is specifically configured to:

[0161] In the point cloud data of the working environment, pixel expansion is performed according to the point cloud data corresponding to the screen pixel coordinates to obtain the target point cloud data within the preset pixel range;

[0162] Based on the target point cloud data within the preset pixel range, the target excavation position is determined from the working environment.

[0163] In an optional implementation, the determination module 702 is specifically configured to:

[0164] Convert the target point cloud data from the camera coordinate system corresponding to the lidar to the pixel coordinate system to obtain the pixel coordinates corresponding to the target point cloud data;

[0165] According to the pixel coordinates corresponding to the target point cloud data, the target excavation position is determined from the working environment.

[0166] In an optional implementation, the determination module 702 is specifically configured to:

[0167] Clustering the pixel coordinates corresponding to the target point cloud data to obtain multiple pixel coordinate categories;

[0168] Calculate the degree of consistency between multiple pixel coordinate categories and screen pixel coordinates respectively;

[0169] determining a target pixel coordinate category from a plurality of pixel coordinate categories according to the degree of agreement;

[0170] According to the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, the target excavation position is determined from the working environment.

[0171] In an optional embodiment, the number of pixel coordinates in the target pixel coordinate category is one;

[0172] The determination module 702 is specifically configured to:

[0173] Convert the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category;

[0174] The target excavation position in the working environment is determined according to the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category.

[0175] In an optional embodiment, the number of pixel coordinates in the target pixel coordinate category is multiple;

[0176] The determination module 702 is specifically configured to:

[0177] Calculate the degree of agreement between multiple pixel coordinates in the target pixel coordinate category and the screen pixel coordinates respectively;

[0178] determining the target pixel coordinates from the plurality of pixel coordinates according to the degree of coincidence;

[0179] According to the point cloud data corresponding to the target pixel coordinates, the target excavation position is determined from the working environment.

[0180] In an optional implementation, the determining module 702 is further configured to:

[0181] Determine the target pixel area on the display screen according to the calibration data, and the target pixel area corresponds to the point cloud data;

[0182] The screen pixel coordinates input for the image of the working environment include: screen pixel coordinates input for the target pixel area.

[0183] In an optional implementation, the determining module 702 is further configured to:

[0184] Determine the target pixel area on the display screen according to the calibration data, and the target pixel area corresponds to the point cloud data;

[0185] A determination module 706 is used to determine whether the target pixel coordinates are within the target pixel area;

[0186] The determination module 702 is specifically configured to:

[0187] If the target pixel coordinates are located in the target pixel area, the point cloud data corresponding to the target pixel coordinates is converted from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the target pixel coordinates;

[0188] According to the three-dimensional coordinates corresponding to the target pixel coordinates, the target excavation position in the working environment is obtained.

[0189] In the excavator control device of this embodiment, a semi-automatic excavation method is provided for the excavator. The operator only needs to input the screen pixel coordinates of the image of the working environment to excavate the target excavation position, realizing the automated operation of digging wherever the key is clicked, effectively reducing the difficulty of operation and improving the working effect. At the same time, it also improves the control accuracy of the excavator. For excavation actions that require simultaneous control of the large and small arms, cockpit, and bucket rotation, it can be completed by simply clicking the mouse on the client interface, effectively lowering the technical threshold of the excavator operator and improving work efficiency.

[0190] The operator only needs to input the screen pixel coordinates of the image of the working environment to excavate the target excavation position, realizing automated operation of digging wherever the button is clicked, effectively reducing the difficulty of operation, improving the working effect, and also improving the control accuracy of the excavator.

[0191] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 12 As shown, the device may include: a processor 801, a memory 802, and a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. When the electronic device is running, the processor 801 communicates with the memory 802 via the bus 803. The processor 801 executes the machine-readable instructions to perform the following steps:

[0192] Acquire an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled;

[0193] Determine a target excavation position of the excavator to be controlled from the working environment based on screen pixel coordinates of an image input of the working environment and point cloud data of the working environment;

[0194] According to the current state parameters, the excavator to be controlled is controlled to perform excavation work at the target excavation position.

[0195] In an optional embodiment, before determining the target excavation position of the excavator to be controlled from the working environment based on the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment, the processor 801 is further configured to:

[0196] Converting the image of the working environment from the image coordinate system to the pixel coordinate system to obtain the pixel coordinates corresponding to the image of the working environment;

[0197] Jointly calibrating the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment to obtain calibration data, where the calibration data is used to indicate a correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment;

[0198] Based on the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment, a target excavation position of the excavator to be controlled is determined from the working environment, including:

[0199] According to the corresponding relationship, the point cloud data corresponding to the screen pixel coordinates is determined from the point cloud data of the working environment;

[0200] Based on the point cloud data corresponding to the screen pixel coordinates, the target excavation location is determined from the working environment.

[0201] In an optional embodiment, when the processor 801 determines the target excavation position from the working environment based on the point cloud data corresponding to the screen pixel coordinates, it is specifically configured to:

[0202] Convert the point cloud data corresponding to the screen pixel coordinates from the camera coordinate system corresponding to the lidar to the world coordinate system to obtain the three-dimensional coordinates corresponding to the screen pixel coordinates;

[0203] Determine the target excavation position in the working environment based on the three-dimensional coordinates corresponding to the screen pixel coordinates.

[0204] In an optional embodiment, when the processor 801 determines the target excavation position from the working environment based on the point cloud data corresponding to the screen pixel coordinates, it is specifically configured to:

[0205] In the point cloud data of the working environment, pixel expansion is performed according to the point cloud data corresponding to the screen pixel coordinates to obtain the target point cloud data within the preset pixel range;

[0206] Based on the target point cloud data within the preset pixel range, the target excavation position is determined from the working environment.

[0207] In an optional embodiment, when the processor 801 determines the target excavation position from the working environment based on the target point cloud data within a preset pixel range, it is specifically configured to:

[0208] Convert the target point cloud data from the camera coordinate system corresponding to the lidar to the pixel coordinate system to obtain the pixel coordinates corresponding to the target point cloud data;

[0209] According to the pixel coordinates corresponding to the target point cloud data, the target excavation position is determined from the working environment.

[0210] In an optional embodiment, when the processor 801 determines the target excavation position from the working environment according to the pixel coordinates corresponding to the target point cloud data, it is specifically configured to:

[0211] Clustering the pixel coordinates corresponding to the target point cloud data to obtain multiple pixel coordinate categories;

[0212] Calculate the degree of consistency between multiple pixel coordinate categories and screen pixel coordinates respectively;

[0213] determining a target pixel coordinate category from a plurality of pixel coordinate categories according to the degree of agreement;

[0214] According to the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, the target excavation position is determined from the working environment.

[0215] In an optional embodiment, the number of pixel coordinates in the target pixel coordinate category is one;

[0216] When the processor 801 determines the target excavation position from the working environment based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, it is specifically configured to:

[0217] Convert the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category;

[0218] The target excavation position in the working environment is determined according to the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category.

[0219] In an optional embodiment, the number of pixel coordinates in the target pixel coordinate category is multiple;

[0220] When the processor 801 determines the target excavation position from the working environment based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, it is specifically configured to:

[0221] Calculate the degree of agreement between multiple pixel coordinates in the target pixel coordinate category and the screen pixel coordinates respectively;

[0222] determining the target pixel coordinates from the plurality of pixel coordinates according to the degree of coincidence;

[0223] According to the point cloud data corresponding to the target pixel coordinates, the target excavation position is determined from the working environment.

[0224] In an optional embodiment, before determining the target excavation position of the excavator to be controlled from the working environment based on the screen pixel coordinates inputted from the image of the working environment and the point cloud data of the working environment, the processor 801 is further configured to:

[0225] Determine the target pixel area on the display screen according to the calibration data, and the target pixel area corresponds to the point cloud data;

[0226] The screen pixel coordinates input for the image of the working environment include: screen pixel coordinates input for the target pixel area.

[0227] In an optional embodiment, before determining the target excavation position from the working environment based on the point cloud data corresponding to the target pixel coordinates, the processor 801 is further configured to:

[0228] Determine the target pixel area on the display screen according to the calibration data, and the target pixel area corresponds to the point cloud data;

[0229] Determine whether the target pixel coordinates are within the target pixel area;

[0230] When the processor 801 determines the target excavation position from the working environment based on the point cloud data corresponding to the target pixel coordinates, it is specifically configured to:

[0231] If the target pixel coordinates are located in the target pixel area, the point cloud data corresponding to the target pixel coordinates is converted from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the target pixel coordinates;

[0232] According to the three-dimensional coordinates corresponding to the target pixel coordinates, the target excavation position in the working environment is obtained.

[0233] In an optional embodiment, the current state parameters include: current state parameters of a plurality of components on the excavator to be controlled;

[0234] When the processor 801 controls the excavator to be controlled to perform an excavation operation at a target excavation location according to the current state parameters, it is specifically configured to:

[0235] A trajectory planner is used to generate the motion trajectory of each component based on its current state parameters and target excavation position;

[0236] According to the motion trajectory of each component, each component is controlled to move so that the excavator to be controlled performs an excavation operation on a target excavation position.

[0237] Through the above method, a semi-automatic excavation method is provided for the excavator. The operator only needs to input the screen pixel coordinates of the image of the working environment to excavate the target excavation position, realizing the automated operation of digging wherever the key is clicked, effectively reducing the difficulty of operation and improving the working effect. At the same time, it also improves the control accuracy of the excavator. For excavation actions that require simultaneous control of the large and small arms, cockpit, and bucket rotation, it can be completed by just clicking the mouse on the client interface, which effectively lowers the technical threshold of the excavator operator and improves work efficiency.

[0238] An embodiment of the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the processor performs the following steps:

[0239] Acquire an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled;

[0240] Determine a target excavation position of the excavator to be controlled from the working environment based on screen pixel coordinates of an image input of the working environment and point cloud data of the working environment;

[0241] According to the current state parameters, the excavator to be controlled is controlled to perform excavation work at the target excavation position.

[0242] In an optional embodiment, before determining the target excavation position of the excavator to be controlled from the working environment based on the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment, the processor is further configured to:

[0243] Converting the image of the working environment from the image coordinate system to the pixel coordinate system to obtain the pixel coordinates corresponding to the image of the working environment;

[0244] Jointly calibrating the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment to obtain calibration data, where the calibration data is used to indicate a correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment;

[0245] Based on the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment, a target excavation position of the excavator to be controlled is determined from the working environment, including:

[0246] According to the corresponding relationship, the point cloud data corresponding to the screen pixel coordinates is determined from the point cloud data of the working environment;

[0247] Based on the point cloud data corresponding to the screen pixel coordinates, the target excavation location is determined from the working environment.

[0248] In an optional embodiment, when the processor determines the target excavation position from the working environment based on the point cloud data corresponding to the screen pixel coordinates, it is specifically configured to:

[0249] Convert the point cloud data corresponding to the screen pixel coordinates from the camera coordinate system corresponding to the lidar to the world coordinate system to obtain the three-dimensional coordinates corresponding to the screen pixel coordinates;

[0250] Determine the target excavation position in the working environment based on the three-dimensional coordinates corresponding to the screen pixel coordinates.

[0251] In an optional embodiment, when the processor determines the target excavation position from the working environment based on the point cloud data corresponding to the screen pixel coordinates, it is specifically configured to:

[0252] In the point cloud data of the working environment, pixel expansion is performed according to the point cloud data corresponding to the screen pixel coordinates to obtain the target point cloud data within the preset pixel range;

[0253] Based on the target point cloud data within the preset pixel range, the target excavation position is determined from the working environment.

[0254] In an optional embodiment, when the processor determines the target excavation position from the working environment based on the target point cloud data within a preset pixel range, it is specifically configured to:

[0255] Convert the target point cloud data from the camera coordinate system corresponding to the lidar to the pixel coordinate system to obtain the pixel coordinates corresponding to the target point cloud data;

[0256] According to the pixel coordinates corresponding to the target point cloud data, the target excavation position is determined from the working environment.

[0257] In an optional embodiment, when the processor determines the target excavation position from the working environment based on the pixel coordinates corresponding to the target point cloud data, it is specifically configured to:

[0258] Clustering the pixel coordinates corresponding to the target point cloud data to obtain multiple pixel coordinate categories;

[0259] Calculate the degree of consistency between multiple pixel coordinate categories and screen pixel coordinates respectively;

[0260] determining a target pixel coordinate category from a plurality of pixel coordinate categories according to the degree of agreement;

[0261] According to the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, the target excavation position is determined from the working environment.

[0262] In an optional embodiment, the number of pixel coordinates in the target pixel coordinate category is one;

[0263] When the processor determines the target excavation position from the operating environment based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, it is specifically used to:

[0264] Convert the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category;

[0265] The target excavation position in the working environment is determined according to the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category.

[0266] In an optional embodiment, the number of pixel coordinates in the target pixel coordinate category is multiple;

[0267] When the processor determines the target excavation position from the operating environment based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category, it is specifically used to:

[0268] Calculate the degree of agreement between multiple pixel coordinates in the target pixel coordinate category and the screen pixel coordinates respectively;

[0269] determining the target pixel coordinates from the plurality of pixel coordinates according to the degree of coincidence;

[0270] According to the point cloud data corresponding to the target pixel coordinates, the target excavation position is determined from the working environment.

[0271] In an optional embodiment, before determining the target excavation position of the excavator to be controlled from the working environment based on the screen pixel coordinates of the image input of the working environment and the point cloud data of the working environment, the processor is further configured to:

[0272] Determine the target pixel area on the display screen according to the calibration data, and the target pixel area corresponds to the point cloud data;

[0273] The screen pixel coordinates input for the image of the working environment include: screen pixel coordinates input for the target pixel area.

[0274] In an optional embodiment, before determining the target excavation position from the working environment based on the point cloud data corresponding to the target pixel coordinates, the processor is further configured to:

[0275] Determine the target pixel area on the display screen according to the calibration data, and the target pixel area corresponds to the point cloud data;

[0276] Determine whether the target pixel coordinates are within the target pixel area;

[0277] When the processor determines the target excavation location from the operating environment based on the point cloud data corresponding to the target pixel coordinates, it is specifically used to:

[0278] If the target pixel coordinates are located in the target pixel area, the point cloud data corresponding to the target pixel coordinates is converted from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the target pixel coordinates;

[0279] According to the three-dimensional coordinates corresponding to the target pixel coordinates, the target excavation position in the working environment is obtained.

[0280] In an optional embodiment, the current state parameters include: current state parameters of a plurality of components on the excavator to be controlled;

[0281] When the processor controls the excavator to be controlled to perform excavation at the target excavation location according to the current state parameters, it is specifically used to:

[0282] A trajectory planner is used to generate the motion trajectory of each component based on its current state parameters and target excavation position;

[0283] According to the motion trajectory of each component, each component is controlled to move so that the excavator to be controlled performs an excavation operation on a target excavation position.

[0284] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment and will not be repeated here.

[0285] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0286] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0287] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0288] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0289] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0290] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for controlling an excavator, characterized in that: include: Acquire an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled; Converting the image of the working environment from an image coordinate system to a pixel coordinate system to obtain pixel coordinates corresponding to the image of the working environment; performing joint calibration on the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment to obtain calibration data, wherein the calibration data is used to indicate a correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment; determining, from the point cloud data of the working environment, point cloud data corresponding to the screen pixel coordinates of the image input of the working environment according to the corresponding relationship; determining a target excavation location from the operating environment based on the point cloud data corresponding to the screen pixel coordinates; According to the current state parameter, the excavator to be controlled is controlled to perform excavation work on the target excavation position.

2. The method according to claim 1, characterized in that Determining a target excavation position from the working environment based on the point cloud data corresponding to the screen pixel coordinates includes: Converting the point cloud data corresponding to the screen pixel coordinates from the camera coordinate system corresponding to the laser radar to the world coordinate system to obtain the three-dimensional coordinates corresponding to the screen pixel coordinates; The target excavation position in the working environment is determined according to the three-dimensional coordinates corresponding to the screen pixel coordinates.

3. The method according to claim 1, characterized in that Determining a target excavation position from the working environment based on the point cloud data corresponding to the screen pixel coordinates includes: In the point cloud data of the working environment, pixel expansion is performed according to the point cloud data corresponding to the screen pixel coordinates to obtain target point cloud data within a preset pixel range; The target excavation position is determined from the working environment according to the target point cloud data within the preset pixel range.

4. The method according to claim 3, characterized in that Determining the target excavation position from the operating environment based on the target point cloud data within the preset pixel range includes: Convert the target point cloud data from the camera coordinate system corresponding to the laser radar to a pixel coordinate system to obtain pixel coordinates corresponding to the target point cloud data; The target excavation position is determined from the working environment according to pixel coordinates corresponding to the target point cloud data.

5. The method according to claim 4, characterized in that Determining the target excavation position from the operating environment according to the pixel coordinates corresponding to the target point cloud data includes: Clustering the pixel coordinates corresponding to the target point cloud data to obtain multiple pixel coordinate categories; respectively calculating the degrees of agreement between the plurality of pixel coordinate categories and the screen pixel coordinates; determining a target pixel coordinate category from the plurality of pixel coordinate categories according to the degree of agreement; The target excavation position is determined from the working environment according to the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category.

6. The method according to claim 5, characterized in that The number of pixel coordinates in the target pixel coordinate category is one; Determining the target excavation position from the operating environment based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category includes: Converting the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category; The target excavation position in the operating environment is determined according to the three-dimensional coordinates corresponding to the pixel coordinates in the target pixel coordinate category.

7. The method according to claim 5, characterized in that There are multiple pixel coordinates in the target pixel coordinate category; Determining the target excavation position from the operating environment based on the point cloud data corresponding to the pixel coordinates in the target pixel coordinate category includes: respectively calculating the degree of agreement between a plurality of pixel coordinates in the target pixel coordinate category and the screen pixel coordinates; determining target pixel coordinates from the plurality of pixel coordinates according to the degree of agreement; The target excavation position is determined from the working environment according to the point cloud data corresponding to the target pixel coordinates.

8. The method according to claim 1, characterized in that Before determining, from the point cloud data of the working environment according to the corresponding relationship, the point cloud data corresponding to the screen pixel coordinates of the image input of the working environment, the method further comprises: determining a target pixel area on the display screen according to the calibration data, wherein the target pixel area corresponds to the point cloud data; The screen pixel coordinates input for the image of the working environment include: the screen pixel coordinates input for the target pixel area.

9. The method according to claim 7, characterized in that Before determining the target excavation position from the working environment based on the point cloud data corresponding to the target pixel coordinates, the method further includes: determining a target pixel area on the display screen according to the calibration data, wherein the target pixel area corresponds to the point cloud data; Determining whether the target pixel coordinates are within the target pixel area; Determining the target excavation position from the operating environment based on the point cloud data corresponding to the target pixel coordinates includes: If the target pixel coordinates are located in the target pixel area, converting the point cloud data corresponding to the target pixel coordinates from the camera coordinate system to the world coordinate system to obtain the three-dimensional coordinates corresponding to the target pixel coordinates; The target excavation position in the working environment is obtained according to the three-dimensional coordinates corresponding to the target pixel coordinates.

10. The method according to claim 1, characterized in that The current state parameters include: current state parameters of multiple components on the excavator to be controlled; The step of controlling the excavator to be controlled to perform excavation work at the target excavation position according to the current state parameter includes: Using a trajectory planner, a motion trajectory of each component is generated according to the current state parameters of each component and the target excavation position; According to the movement trajectory of each component, each component is controlled to move, so that the excavator to be controlled performs the excavation operation on the target excavation position.

11. An excavator control device, characterized in that: include: an acquisition module, configured to acquire an image of the working environment of the excavator to be controlled captured by a camera, point cloud data of the working environment captured by a laser radar, and current state parameters of the excavator to be controlled; a conversion module, configured to convert the image of the working environment from an image coordinate system to a pixel coordinate system, and obtain pixel coordinates corresponding to the image of the working environment; a calibration module, configured to jointly calibrate the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment to obtain calibration data, wherein the calibration data is used to indicate a correspondence between the pixel coordinates corresponding to the image of the working environment and the point cloud data of the working environment; a determination module, configured to determine, from the point cloud data of the working environment, point cloud data corresponding to the screen pixel coordinates of the image input of the working environment according to the corresponding relationship; The determination module is further configured to determine a target excavation position from the operating environment based on the point cloud data corresponding to the screen pixel coordinates; A control module is used to control the excavator to be controlled to perform an excavation operation on the target excavation position according to the current state parameter.

12. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to execute the excavator control method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the excavator control method according to any one of claims 1 to 10 is executed.

Images