Method, system, device and electronic equipment for remotely controlling an excavator

Abstract

The present application discloses a method, system, device, electronic device and computer-readable storage medium for remotely controlling an excavator, the method comprising: receiving an excavation request sent by a remote control terminal, and acquiring operation scene point cloud data according to the excavation request; sending the operation scene point cloud data to the remote control terminal, so that the remote control terminal can construct an operation scene point cloud model according to the operation scene point cloud data; receiving an excavation instruction sent by the remote control terminal based on the operation scene point cloud model, and controlling the excavator to perform a target excavation action according to the excavation instruction. The method introduces the concept of point cloud model into the remote control of the excavator, and provides more scene information to the remote operator through the operation scene point cloud model, which helps the remote operator to accurately control the remote operation of the excavator. The method solves the problem of low remote control accuracy and low excavation efficiency due to the limited scene information provided by the operation scene image in the prior art.

Description

Technical Field

[0001] The present application relates to the technical field of remote control of excavators, and in particular to a method for remotely controlling an excavator. The present application also relates to an excavator remote control system, an excavator, an excavator remote control device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of computer remote control technology, remote control of excavators has also become an important research topic. Excavator remote control mainly displays the working scene image of the excavator on the screen of the remote control terminal, so that the remote operator can control the excavator operation based on the working scene image at the remote control terminal.

[0003] The existing remote control of excavators is based on two-dimensional images of the working scene. The remote operator needs to rely on rich experience to predict the location and depth of excavation. This often leads to misjudgment by the remote operator, resulting in wrong digging, empty digging, etc. during the excavator operation. Therefore, the existing remote control method of excavators has the problem of low remote control accuracy and low excavation efficiency due to the limited scene information provided by the working scene image. Summary of the invention

[0004] The present application provides a method, system, device, excavator, electronic device and computer-readable storage medium for remotely controlling an excavator to solve the problem of low remote control accuracy and low excavation efficiency due to limited scene information provided by the working scene image in the existing method of remotely controlling an excavator.

[0005] The present application embodiment provides a method for remotely controlling an excavator, the method being applied to an excavator end, the method comprising:

[0006] Receiving a mining request sent by a remote control terminal, and acquiring point cloud data of an operation scene according to the mining request;

[0007] Sending the operation scene point cloud data to the remote control terminal, so that the remote control terminal constructs an operation scene point cloud model according to the operation scene point cloud data;

[0008] Receive the excavation instruction sent by the remote control terminal based on the work scene point cloud model, and control the excavator to perform the target excavation action according to the excavation instruction.

[0009] The embodiment of the present application also provides a method for remotely controlling an excavator, the method being applied to a remote control terminal, the remote control terminal comprising at least a display screen, the method comprising:

[0010] Sending an excavation request to the excavator, so that the excavator obtains the point cloud data of the operation scene according to the excavation request;

[0011] Receiving the operation scene point cloud data sent by the excavator, constructing an operation scene point cloud model according to the operation scene point cloud data, and displaying the operation scene point cloud model on the display screen;

[0012] Based on the point cloud model of the operation scene, an excavation instruction is sent to the excavator end, so that the excavator end controls the excavator to perform a target excavation action according to the excavation instruction.

[0013] The embodiment of the present application also provides an excavator remote control system, the system comprising: a remote control terminal and an excavator terminal;

[0014] The remote control end is used to send an excavation request to the excavator end so that the excavator end obtains the work scene point cloud data according to the excavation request; receives the work scene point cloud data sent by the excavator end, constructs a work scene point cloud model according to the work scene point cloud data, and displays the work scene point cloud model on a display screen; based on the work scene point cloud model, sends an excavation instruction to the excavator end, so that the excavator end controls the excavator to perform a target excavation action according to the excavation instruction.

[0015] The excavator end is used to receive the excavation request sent by the remote control end, and obtain the work scene point cloud data according to the excavation request; send the work scene point cloud data to the remote control end so that the remote control end can construct the work scene point cloud model according to the work scene point cloud data; receive the excavation instruction sent by the remote control end based on the work scene point cloud model, and control the excavator to perform the target excavation action according to the excavation instruction.

[0016] The embodiment of the present application further provides an excavator, the excavator is equipped with a vehicle-mounted system, the vehicle-mounted system at least includes: a first receiving component, a data acquisition component, a data sending component, a second receiving component, and a control component;

[0017] The first receiving component is used to receive a mining request sent by a remote control terminal;

[0018] The data acquisition component is used to acquire the operation scene point cloud data according to the mining request;

[0019] The data sending component is used to send the operation scene point cloud data to the remote control terminal;

[0020] The second receiving component is used to receive the mining instruction sent by the remote control terminal based on the operation scene point cloud model;

[0021] The control component is used to control the excavator to perform a target excavation action according to the excavation instruction.

[0022] The embodiment of the present application also provides an excavator remote control device, which is applied to the excavator end, and includes: a point cloud data acquisition unit, a point cloud data sending unit, and an excavation action control unit;

[0023] The point cloud data acquisition unit is used to receive a mining request sent by a remote control terminal, and acquire the point cloud data of the operation scene according to the mining request;

[0024] The point cloud data sending unit is used to send the operation scene point cloud data to the remote control terminal, so that the remote control terminal can construct an operation scene point cloud model according to the operation scene point cloud data;

[0025] The excavation action control unit is used to receive the excavation instructions sent by the remote control end based on the work scene point cloud model, and control the excavator to perform the target excavation action according to the excavation instructions.

[0026] The embodiment of the present application also provides a remote control device for an excavator, the device is applied to a remote control terminal, the remote control terminal includes at least a display screen, the device includes: an excavation request sending unit, a point cloud model building unit, and an excavation instruction sending unit;

[0027] The mining request sending unit is used to send a mining request to the excavator end, so that the excavator end obtains the operation scene point cloud data according to the mining request;

[0028] The point cloud model construction unit is used to receive the operation scene point cloud data sent by the excavator, construct an operation scene point cloud model according to the operation scene point cloud data, and display the operation scene point cloud model on the display screen;

[0029] The excavation instruction sending unit is used to send an excavation instruction to the excavator side based on the work scene point cloud model, so that the excavator side controls the excavator to perform a target excavation action according to the excavation instruction.

[0030] The embodiment of the present application also provides an electronic device, including: a memory and a processor;

[0031] The memory is used to store one or more computer instructions;

[0032] The processor is used to execute the one or more computer instructions to implement the above method.

[0033] The embodiment of the present application also provides a computer-readable storage medium, on which one or more computer instructions are stored. When the instructions are executed by a processor, the above method is executed.

[0034] Compared with the prior art, the method for remotely controlling an excavator provided in the present application comprises: receiving an excavation request sent by a remote control terminal, and obtaining operation scene point cloud data according to the excavation request; sending the operation scene point cloud data to the remote control terminal, so that the remote control terminal can construct an operation scene point cloud model according to the operation scene point cloud data; receiving an excavation instruction sent by the remote control terminal based on the operation scene point cloud model, and controlling the excavator to perform a target excavation action according to the excavation instruction. By acquiring the operation scene point cloud data and constructing the operation scene point cloud model, the method enables the remote operator to accurately issue an excavation instruction based on the operation scene point cloud model at the excavator remote control terminal, thereby improving the accuracy of remote control and the excavation efficiency. The method for remotely controlling an excavator provided in the present application introduces the concept of point cloud model into the remote control of the excavator, and provides more scene information for the remote operator through the point cloud model of the excavator operation scene, which helps the remote operator to accurately control the remote operation of the excavator according to the operation scene point cloud model. The method solves the problem that the existing method for remotely controlling an excavator has low remote control accuracy and low excavation efficiency due to the limited scene information provided by the operation scene image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is an application system diagram of a method for remotely controlling an excavator provided in an embodiment of the present application;

[0036] Figure 2 is a flow chart of a method for remotely controlling an excavator provided in a first embodiment of the present application;

[0037] Figure 3 It is a schematic diagram of a method for obtaining pixel coordinate information corresponding to a point provided in the first embodiment of the present application;

[0038] Figure 4 is a schematic diagram of a method for obtaining depth information corresponding to a point provided in the first embodiment of the present application;

[0039] Figure 5 is a schematic diagram of a method for obtaining current posture information of an excavator provided in the first embodiment of the present application;

[0040] Figure 6 is a flow chart of a method for remotely controlling an excavator provided in a second embodiment of the present application;

[0041] Figure 7 is a schematic diagram of a method for selecting a target viewing angle provided in a second embodiment of the present application;

[0042] Figure 8 is a schematic diagram of a method for switching a target viewing angle provided in a second embodiment of the present application;

[0043] Fig. 9 is a schematic diagram of a method for selecting a target excavation point provided in a second embodiment of the present application;

[0044] Fig.10 is a signaling flow chart of a method for remotely controlling an excavator provided in a second embodiment of the present application;

[0045] Fig.11 is a structural schematic diagram of an excavator remote control system provided in a third embodiment of the present application;

[0046] Fig.12 is a structural schematic diagram of an excavator provided in a fourth embodiment of the present application;

[0047] Fig.13 is a structural schematic diagram of an excavator remote control device provided in a fifth embodiment of the present application;

[0048] Fig.14 is a structural schematic diagram of an excavator remote control device provided in a sixth embodiment of the present application;

[0049] Fig.15 It is a schematic diagram of the structure of an electronic device provided in the seventh embodiment of the present application. DETAILED DESCRIPTION

[0050] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0051] The professional terms involved in the embodiments of the present application are explained below to facilitate understanding.

[0052] Remote control refers to the process of a remote operator controlling a remote device through a computer network. For example, remote control of a computer is the process of a remote operator operating a remote computer through a computer network, such as a remote operator controlling a remote computer to help a remote user solve computer problems. Another example is remote control of an excavator, which is the process of a remote operator using the control components of the remote operation terminal to operate an excavator at the construction site through a computer network, such as remotely controlling an excavator for mining.

[0053] The world coordinate system is the absolute coordinate system of the system. Before the user coordinate system is established, the coordinates of all points on the image are determined by the origin of this coordinate system.

[0054] The camera coordinate system takes the optical center of the camera as its origin, the x-axis and y-axis are parallel to the X-axis and Y-axis of the image, and the z-axis is the optical axis of the camera, which is perpendicular to the graphic plane.

[0055] The pixel coordinate system is a direct coordinate system uv established with the upper left corner of the image as the origin and with pixels as the unit. The horizontal coordinate u and vertical coordinate v of a pixel represent the column and row number of the pixel in the image array respectively.

[0056] Point cloud is a collection of points. Each point in the point cloud is a three-dimensional point. The information contained in the point cloud includes the location information of each point (that is, the three-dimensional coordinates X, Y, Z of each point), as well as color, light intensity, category label, normal vector, grayscale value, etc. Point cloud can atomize the real world and restore the real world through high-precision point cloud data.

[0057] Point cloud data refers to a set of vectors in a three-dimensional coordinate system. These vectors are usually expressed in the form of (X, Y, Z) three-dimensional coordinates and are generally used to represent the outer surface shape of an object. In addition to the geometric position information represented by (X, Y, Z), point cloud data can also represent the color information and gray value information of a point. i ={X i , Y i , Z i , ...} represents a point in the point cloud, then Point Cloud = {P 1 , P 2 , P 3 , ..., P i} can represent a set of point cloud data.

[0058] The point cloud model is a model formed by converting point cloud data into a 3D surface through surface reconstruction methods. It is a three-dimensional model obtained by rendering the obtained point cloud data.

[0059] Excavators are an indispensable piece of engineering equipment in engineering construction, and they play an irreplaceable role in the fields of construction, transportation, and mining. At present, most of the operations of excavators are still completed by operators controlling the excavators. The working environment of excavators is often harsh, with a lot of dust, and there is a risk of disasters such as landslides and mudslides. If operators are in this environment for a long time, it will not only seriously affect their health, but also pose a great threat to their personal safety.

[0060] With the rapid development of remote control technology, remote control of excavators has also received attention. Remote control of excavators is to display the working scene image of the excavator on the screen of the remote control terminal, so that the remote operator can control the excavator operation based on the working scene image at the remote control terminal. Therefore, image acquisition is a crucial step in excavator remote control technology. The more scene information the working scene image includes, the more accurately the remote operator can remotely control the excavator.

[0061] At present, the work scene image based on which the excavator is remotely controlled is a two-dimensional image, which can only provide the remote operator with the location information of each point in the work scene, but cannot provide the depth information of each point, so the remote operator needs to rely on rich excavation experience to predict the excavation location and depth. This often leads to misjudgment by the remote operator, resulting in wrong excavation, empty excavation, etc. during the excavator operation, and in severe cases, even causing the excavator to overturn. Therefore, the existing method of remotely controlling the excavator has the problem of low remote control accuracy and low excavation efficiency due to the limited scene information provided by the work scene image.

[0062] In addition, the existing remote control method of the excavator requires the remote operator to control the excavator operation through simulated driving in the simulated cockpit of the remote control terminal, which has extremely high requirements for the establishment of the remote control terminal and the business requirements of the remote operator. Therefore, although the existing remote control method of the excavator improves the operator's working environment, it does not reduce the operator's labor intensity and business threshold.

[0063] Therefore, the existing remote control methods for excavators have the problems of low remote control accuracy and low excavation efficiency due to the limited scene information provided by the working scene image. At the same time, there are problems of high labor intensity and high business threshold for remote operators.

[0064] In view of this, the present application provides a method for remotely controlling an excavator. The method acquires the point cloud data of the operation scene and constructs a point cloud model of the operation scene, so that the remote operator can accurately issue excavation instructions based on the point cloud model of the operation scene at the remote control end of the excavator, thereby improving the accuracy of remote control and the excavation efficiency. At the same time, the method for remotely controlling an excavator provided by the present application enables the remote operator to control the excavator to excavate the target excavation point by clicking on the target excavation point on the point cloud model of the operation scene, which greatly reduces the labor intensity of the remote operator and the business threshold.

[0065] The method for remotely controlling an excavator, the remote control system, the excavator, the remote control device, the electronic device, and the computer-readable storage medium described in the present application are further described in detail below in conjunction with specific embodiments and drawings.

[0066] Figure 1 It is an application system diagram of a method for remotely controlling an excavator provided in an embodiment of the present application.

[0067] like Figure 1 As shown, the application system includes: an excavator end 101 and a remote control end 102. The excavator end 101 is communicatively connected to the remote control end 102 via a network. The excavator end 101 is located in an engineering construction scene, and at least includes an excavator and a vehicle-mounted system configured on the excavator. The vehicle-mounted system is deployed with the method for remotely controlling an excavator provided in the present application, and provides the remote control end 102 with the operating scene point cloud data of the excavator. The remote control end 102 is located outside the engineering construction scene, and at least includes a display screen, on which a working scene point cloud model constructed by the working scene point cloud data is displayed. The remote operator controls the excavator to perform target excavation operations in the engineering construction scene according to the working scene point cloud model displayed on the display screen.

[0068] A first embodiment of the present application provides a method for remotely controlling an excavator, which is applied to an excavator end, where the excavator end is an excavator located in a working scene.

[0069] Figure 2 Flowchart of the method for remotely controlling an excavator provided in this embodiment. Figure 2 The method for remotely controlling an excavator provided by this embodiment is described in detail. The embodiments described below are used to explain the technical solution of this application and are not intended to be used as limitations for actual use.

[0070] like Figure 2 As shown, the method for remotely controlling an excavator provided in this embodiment includes the following steps:

[0071] Step S201, receiving a mining request sent by a remote control terminal, and acquiring point cloud data of an operation scene according to the mining request.

[0072] The excavation request may be understood as an excavation start request sent by a remote operator to an excavator via a remote control terminal, which may include target viewing angle selection information.

[0073] The viewing angle can be understood as the angle range of the camera receiving the image, which can also be called the field of view, describing the angle at which the camera captures the image. For the same scene, different scene images will be seen from different viewing angles. The viewing angle described in this embodiment is the image capture angle of the working scene in front of the excavator.

[0074] The target viewing angle can be understood as the desired image capture angle when the remote operator views the operation scene. In an optional implementation of this embodiment, the target viewing angle includes a fixed viewing angle of the operation scene and an omnidirectional viewing angle of the operation scene.

[0075] The fixed viewing angle of the working scene may include one image capturing angle of the working scene, or may include multiple image capturing angles of the working scene. The fixed viewing angle of the working scene provided in this embodiment includes at least one of the following viewing angles: a front viewing angle of the working scene, a top viewing angle of the working scene, a left viewing angle of the working scene, and a right viewing angle of the working scene.

[0076] The working scene under the formal perspective is the front view image of the working scene, the working scene under the overhead perspective is the overhead image of the working scene, the working scene under the left-side perspective is the left-side view image of the working scene, and the working scene under the right-side perspective is the right-side view image of the working scene.

[0077] The omnidirectional perspective of the work scene can be understood as a 360-degree image capture perspective of the work scene. The work scene under the omnidirectional perspective is a 360-degree image of the work scene, that is, the image of the work scene under any perspective can be viewed through the omnidirectional perspective.

[0078] In an optional implementation of this embodiment, in response to the excavator side receiving a mining request including target perspective selection information sent by the remote control side, the excavator side will obtain the work scene point cloud data under the target perspective based on the mining request including the target perspective selection information.

[0079] The point cloud data can be understood as a set of vectors in a three-dimensional coordinate system. Each vector represents a point in the point cloud and can be expressed in the form of three-dimensional coordinates. i ={X i , Y i , Z i} represents a point in the point cloud, then Point Cloud = {P 1 , P 2 , P 3 , ..., P i} can represent a set of point cloud data.

[0080] In this embodiment, the operation scene point cloud data can be understood as a set of vectors corresponding to each point in the operation scene. The vector corresponding to each point is a vector composed of point information corresponding to each point, including pixel coordinate information and depth information corresponding to each point in the operation scene. The pixel coordinate information can be understood as corresponding to {X i , Y i}, the depth information can be understood as corresponding to {Z i}, therefore, the point cloud data including the pixel coordinate information and depth information corresponding to each point is the one-dimensional coordinate point corresponding to {X i , Y i , Z i} is a set of three-dimensional coordinate points.

[0081] This embodiment provides an optional implementation method for obtaining point cloud data of an operation scene, including the following steps:

[0082] First, the original image of the working scene from the target perspective is collected by the binocular camera set at the front end of the excavator body.

[0083] The binocular camera is set at the front of the excavator body, and can capture images of the working scene at the front of the excavator body at any perspective. The binocular camera imitates the two eyes to shoot the same scene, and can obtain images of the same scene at different perspectives. By comparing the two images, the same feature points in the two images are extracted, and these feature points are used to perform stereo matching on the two images to obtain the correspondence between all points in the two images. Then, based on the known relevant parameters of the binocular camera, the depth of all points in the image in the actual space can be calculated.

[0084] Second, the coordinate information of each point in the original image of the work scene is converted from the world coordinate system to the pixel coordinate system to obtain the pixel coordinate information corresponding to each point in the original image of the work scene.

[0085] The pixel coordinate information corresponding to each point in the original image of the work scene needs to be obtained through coordinate system conversion, which involves four coordinate systems, namely, the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. The acquisition of pixel coordinate information includes the first conversion from the world coordinate system to the camera coordinate system, the second conversion from the camera coordinate system to the image coordinate system, and the third conversion from the image coordinate system to the pixel coordinate system. The first conversion is the conversion of three-dimensional information to three-dimensional information, the second conversion is the conversion of three-dimensional information to two-dimensional information, and the third conversion is the conversion of two-dimensional information to two-dimensional information. The following describes the coordinate system conversion method of point coordinate information.

[0086] Figure 3 It is a schematic diagram of a method for obtaining pixel coordinate information corresponding to a point provided in this embodiment.

[0087] like Figure 3 As shown, the method for obtaining pixel coordinate information corresponding to a point provided in this embodiment includes:

[0088] S31, conversion from world coordinate system to camera coordinate system.

[0089] like Figure 3 As shown in (a), (X w ,Y w ,Z w ) is the coordinate information of the target point in the world coordinate system, (X c ,Y c ,Z c ) is the point coordinate information of the target point in the camera coordinate system, then the conversion formula of the target point from the world coordinate system to the camera coordinate system is:

[0090]

[0091] Among them, R represents the rotation matrix from the world coordinate system to the camera coordinate system, T represents the translation variable, and the transformation from the world coordinate system to the camera coordinate system only transforms the reference coordinate system and does not involve projection transformation.

[0092] S32, conversion from camera coordinate system to image coordinate system.

[0093] like Figure 3 As shown in (b), (x c ,y c ,z c ) is the target point coordinate information in the camera coordinate system, and (x, y) is the target point coordinate information in the image coordinate system. The image coordinate system has no z-axis component, so the conversion of the target point from the camera coordinate system to the image coordinate system involves projection transformation. The projection transformation involves the camera focal length f and the scale factor, which is related to z. c Related. Figure 3 As shown in (b), the camera coordinate system and the image coordinate system have the following projection relationship:

[0094] ΔABO c ~ΔoCO c

[0095] ΔPBO c ~ΔpCO c

[0096] Therefore, the conversion formula of the target point from the camera coordinate system to the image coordinate system is:

[0097]

[0098] S33, conversion from the image coordinate system to the pixel coordinate system.

[0099] like Figure 3 As shown in (c), (x, y) is the coordinate information of the target point in the image coordinate system, (u 0 ,v 0) is the coordinate information of the target point in the pixel coordinate system. In the image coordinate system, the origin of the coordinate system is in the middle of the image, and in the pixel coordinate system, the origin of the coordinate system is in the upper left corner of the image. Figure 3 As shown in (c), the image coordinate system and the pixel coordinate system have the following corresponding relationship:

[0100]

[0101] Therefore, the conversion formula of the target point from the image coordinate system to the pixel coordinate system is:

[0102]

[0103] Third, calculate the depth information of each point in the original image of the work scene.

[0104] The acquisition of depth information involves stereo matching of two images of the same scene obtained by a binocular camera from different perspectives. Figure 4 It is a schematic diagram of a method for obtaining depth information corresponding to a point provided in this embodiment.

[0105] like Figure 4 As shown, O L and O R are the optical centers of the two cameras of the binocular camera respectively. The line connecting the two optical centers is called the baseline, and the length of the baseline is b. The plane where the optical centers are located is called the projection plane, f is the focal length of the camera, and the target point P in the three-dimensional space is imaged at point P in the left and right cameras respectively. L and P R , line segment X L and X R are the distances from the imaging points of the left and right cameras to the left imaging plane, respectively. Then the parallax of the target point P between the left and right cameras can be defined as d = |X L -X R |.

[0106] Two imaging points P L and P R The distance between them is:

[0107]

[0108] The relationship between parallax and depth can be obtained through the principle of triangulation. L O R With triangle PP L P R Similarly, the relationship between depth and parallax can be expressed as:

[0109]

[0110] Therefore, the distance from the target point P to the center of the projection plane can be expressed as:

[0111]

[0112] The distance from the target point P to the center of the projection plane is the depth information of the target point P.

[0113] Fourth, based on the acquired pixel coordinate information and depth information corresponding to each point, a work scene point cloud data including the pixel coordinate information and depth information corresponding to each point in the work scene is formed.

[0114] By converting the point coordinate system and calculating the point depth, the pixel coordinate information and depth information corresponding to each point in the original image of the work scene can be obtained, thus forming the point cloud data of the work scene.

[0115] Step S202: sending the operation scene point cloud data to the remote control terminal, so that the remote control terminal constructs an operation scene point cloud model according to the operation scene point cloud data.

[0116] The excavator side sends the acquired action scene point cloud data to the remote control side, and the remote control side can construct a corresponding action scene point cloud model based on the action scene point cloud data.

[0117] Generally, the amount of point cloud data obtained is large, and it is necessary to transmit the point cloud data in slices according to the transmission bandwidth between the excavator and the remote control end. When the remote control end receives all the point cloud data slices, it can use point cloud rendering technology to build a point cloud model.

[0118] The work scene point cloud data includes the pixel coordinate information and depth information corresponding to each point in the work scene. Therefore, the work scene point cloud model constructed by the remote control terminal based on the pixel coordinate information and depth information corresponding to each point in the work scene will also include the pixel coordinate information corresponding to each point in the work scene and the depth information corresponding to each point in the work scene.

[0119] In an optional implementation of the present embodiment, the remote control end sends a mining request including target perspective selection information to the excavator end, and the excavator end obtains the work scene point cloud data under the target perspective, and sends the work scene point cloud data under the target perspective to the remote control end, so that the remote control end can construct the work scene point cloud model under the target perspective based on the work scene point cloud data under the target perspective.

[0120] The image capture perspective of the work scene affects the information included in the work scene point cloud data, and further affects the information included in the work scene point cloud model. For example: the image capture perspective of the work scene is the left view perspective of the work scene, then the work scene point cloud data obtained by the excavator end includes the pixel coordinate information and depth information corresponding to each point in the left view of the work scene, and correspondingly, the work scene point cloud model constructed by the remote control end also includes the pixel coordinate information and depth information corresponding to each point in the left view of the work scene.

[0121] Step S203, receiving the excavation instruction sent by the remote control terminal based on the point cloud model of the operation scene, and controlling the excavator to perform the target excavation action according to the excavation instruction.

[0122] The excavation instruction may be understood as a command sent by a remote operator to an excavator via a remote control terminal to control the excavator to perform an excavation action, which may include pixel coordinate information and depth information corresponding to a target excavation point.

[0123] The target excavation point can be understood as the expected excavation point selected by the remote operator based on the point cloud model of the operation scene. Through the remote operator's selection of the target excavation point, the excavator can obtain the pixel coordinate information and depth information corresponding to the target excavation point, and then control the excavator to excavate the target excavation point according to the pixel coordinate information and depth information corresponding to the target excavation point.

[0124] In an optional implementation provided by this embodiment, before the step of controlling the excavator to excavate the target excavation point, the following steps are also included:

[0125] First, in response to receiving the excavation instruction sent by the remote control end based on the point cloud model of the work scene, the current posture state parameters of the excavator are obtained, and the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and the current posture information of the excavator.

[0126] The posture information mainly includes the angular position information between the excavator's boom and cabin, the forearm and boom, the bucket and forearm, and the cabin and tracks. The posture information of the excavator can be obtained by installing posture sensors on the excavator's boom, forearm, bucket, cabin and other parts to obtain the joint angles of the boom, forearm, bucket and cabin.

[0127] The current posture information may be understood as posture information of the excavator in a waiting-to-start state or a current working state.

[0128] Figure 5 It is a schematic diagram of the method for obtaining the current posture information of the excavator provided in this embodiment.

[0129] like Figure 5 As shown, the method for obtaining the current posture information mainly includes:

[0130] S51, define the coordinate systems of various components of the excavator, specifically defined as: the excavator overall coordinate system base_link, the cockpit coordinate system upper_body_link, the boom root coordinate system arm_base_link, the arm coordinate system boom_link, the arm coordinate system arm_link, and the bucket coordinate system gripper_link. The boom root coordinate system arm_base_link and the cockpit coordinate system upper_body_link are in a fixed conversion relationship.

[0131] S52, obtaining, through the attitude sensor, the conversion relationship between the arm base coordinate system arm_base_link and the arm coordinate system boom_link, and the conversion relationship between the arm coordinate system boom_link and the forearm coordinate system arm_link.

[0132] Depend on Figure 5 It can be seen that the boom coordinate system boom_link, the arm coordinate system arm_link, and the bucket coordinate system gripper_link rotate around the y-axis. The boom, arm, and bucket move downward as a counterclockwise rotation around the y-axis, and the joint angles of the boom, arm, and bucket gradually increase. The boom, arm, and bucket move upward as a clockwise rotation around the y-axis, and the joint angles of the boom, arm, and bucket gradually decrease. The cockpit coordinate system upper_body_link rotates around the z-axis. The cockpit moves to the left as a counterclockwise rotation around the z-axis, and the joint angle of the cockpit gradually increases. The cockpit moves to the right as a clockwise rotation around the z-axis, and the joint angle of the cockpit gradually decreases.

[0133] S53, define the joint zero position, specifically defined as: the boom coordinate system boom_link and the boom root coordinate system arm_base_link form an angle of 1 radian, the forearm coordinate system arm_link and the boom coordinate system boom_link form an angle of 1.5 radians, and the bucket coordinate system gripper_link is parallel to the forearm coordinate system arm_link.

[0134] S54, based on the definition of the joint zero position, the current posture information of the excavator can be obtained through the sensor. For example, if the sensor senses that the forearm rotates upward by 1 arc based on the joint zero position, the current posture information includes that the joint angle between the forearm and the upper arm is 0.5 arc.

[0135] The pixel coordinate information corresponding to the current location of the excavator can be obtained according to the method for obtaining the pixel coordinate information corresponding to the point provided in step S201.

[0136] By obtaining the pixel coordinate information corresponding to the current position of the excavator and the current posture information of the excavator, the current posture state parameters of the excavator can be obtained.

[0137] In order to facilitate understanding of the information content included in the current posture state parameters of the excavator, an example is given in the form of a table. For example, the excavator is currently at point A, and the current posture of the excavator is M posture. Then the current posture state parameters of the excavator include the pixel coordinate information (30, 23) of point A and the posture information (0.5, 1.5, 2, 0) of posture M, where 30 is the horizontal coordinate information of point A in the pixel coordinate system, in pixels, 23 is the vertical coordinate information of point A in the pixel coordinate system, in pixels, 0.5 is the joint degree between the bucket and the forearm in the M posture, in radians, 1.5 is the joint degree between the forearm and the upper arm in the M posture, in radians, 2 is the joint degree between the upper arm and the cabin in the M posture, in radians, and 0 is the joint degree between the cabin and the crawler in the M posture, in radians. The current posture state parameters of the excavator can be shown in Table 1:

[0138] Table 1 Current posture state parameters of the excavator

[0139]

[0140] The current posture state parameters of the excavator can also be expressed in vector form, such as {30, 23, 0.5, 1.5, 2.0, 0}. The specific expression form of the current posture state parameters of the excavator is not limited here.

[0141] Second, according to the pixel coordinate information and depth information corresponding to the target excavation point, the target posture state parameters of the excavator are calculated, and the target posture state parameters include the pixel coordinate information corresponding to the target excavation point and the target posture information of the excavator.

[0142] Through the remote operator's selection of the target excavation point, the excavator end can obtain the pixel coordinate information and depth information corresponding to the target excavation point, and the target posture information of the excavator can be calculated based on the pixel coordinate information and depth information corresponding to the target excavation point. For example, based on the depth information corresponding to the target excavation point, it is found that when the excavator is in the joint zero position, it cannot access the operating content of the target excavation point. Then, it is necessary to move the boom, forearm, and bucket counterclockwise downward around the y-axis, and the corresponding angular joints will increase. If the boom is moved downward by 2 arcs and the forearm is moved downward by 1 arc, the operating content of the target excavation point can be accessed. Then, the target posture information includes a joint angle of 3 arcs between the boom and the root of the boom, and a joint angle of 2.5 arcs between the forearm and the boom.

[0143] Since the pixel coordinate information corresponding to the target excavation point has been obtained, the target posture state parameters of the excavator can be obtained by calculating the target posture information of the excavator.

[0144] If the target excavation point is point B, the excavator can obtain the corresponding pixel coordinate information (40, 10) and depth information (2) of point B, where 40 is the horizontal coordinate information of point B in the pixel coordinate system, the unit is pixel, 10 is the vertical coordinate information of point B in the pixel coordinate system, the unit is pixel, and 2 is the depth information of point B, the unit is meter.

[0145] According to the depth information of point B of 2 meters and the posture information of the excavator's current posture M (0.5, 1.5, 2, 0), it can be known that after moving the excavator in posture M to point B, move the upper arm 2 arcs counterclockwise downward around the y-axis, move the lower arm 0.5 arcs counterclockwise downward around the y-axis, move the bucket 1 arc counterclockwise downward around the y-axis, and rotate the cabin 30 arcs counterclockwise around the z-axis to access the target excavation point.

[0146] Then, the posture information of the target posture N of the excavator is (2.5, 2.0, 3, 30), and the target posture state parameters of the excavator can be shown in Table 2:

[0147] Table 2 Target posture state parameters of excavator

[0148]

[0149] The target posture state parameters of the excavator can also be expressed in vector form, such as {40, 10, 2.5, 2.0, 3.0, 30}. The specific expression form of the target posture state parameters of the excavator is not limited here.

[0150] Third, according to the current posture state parameters and the target posture state parameters, the motion trajectory of the excavator is calculated, and the motion trajectory includes the point movement trajectory of the excavator and the posture change trajectory of the excavator.

[0151] Based on the current posture state parameters obtained above and the calculated target posture state parameters, the trajectory planner can be used in combination with the kinematic equations to calculate the excavator's point movement trajectory and the excavator's posture change trajectory.

[0152] The point movement trajectory can be understood as the movement trajectory of the excavator from the current point to the target excavation point, which is mainly calculated through the pixel coordinate information corresponding to the current point of the excavator and the pixel coordinate information corresponding to the target excavation point. It can be a straight line trajectory between the current point and the target excavation point.

[0153] The posture change trajectory can be understood as the movement trajectory of the excavator components when the excavator's posture is converted from the current posture information to the target posture information, which is mainly calculated through the excavator's current posture information and the target posture information, and can be the movement trajectory of the various components of the excavator that are comprehensively optimized.

[0154] After the motion trajectory of the excavator is obtained, the excavator can be controlled to move to the target excavation point according to the point movement trajectory, and the posture of the excavator can be changed according to the posture change trajectory to excavate the target excavation point.

[0155] If the current posture state information (A, M) of the excavator is {30, 23, 0.5, 1.5, 2.0, 0}, and the target posture state information (B, N) is {40, 10, 2.5, 2.0, 3.0, 30}, then the movement of the excavator from point A to point B is the point movement trajectory, and the change from posture M to posture N is the posture change trajectory. The motion trajectory of the excavator can be shown in Table 3:

[0156] Table 3 Movement trajectory of the excavator

[0157]

[0158]

[0159] According to the motion trajectory shown in Table 3, the excavator moves from point A to point B, and at point B, changes from posture M to posture N, thereby completing the target excavation action at the target excavation point.

[0160] The above-mentioned first embodiment takes the excavator end as the main body and describes in detail the method for remotely controlling an excavator provided by the present application. The method for remotely controlling an excavator provided by the present application includes but is not limited to the method described in the first embodiment of the present application.

[0161] The second embodiment of the present application provides a method for remotely controlling an excavator, which is applied to a remote control terminal, which is a computer, mobile phone, tablet computer, etc. located outside the working scene, and the remote control terminal includes at least one display screen.

[0162] Figure 6 It is a flow chart of the method for remotely controlling an excavator provided in this embodiment.

[0163] like Figure 6 As shown, the method for remotely controlling an excavator provided in this embodiment includes the following steps:

[0164] Step S601, sending an excavation request to the excavator side, so that the excavator side obtains the operation scene point cloud data according to the excavation request.

[0165] The excavation request may be understood as an excavation start request sent by a remote operator to the excavator via a remote control terminal.

[0166] The mining request provided in this embodiment may include target perspective selection information, and the target perspective may be understood as the desired image capture angle when the remote operator views the operation scene. In an optional implementation of this embodiment, the target perspective includes a fixed perspective of the operation scene and an omnidirectional perspective of the operation scene.

[0167] The fixed viewing angle of the working scene may include one image capturing angle of the working scene, or may include multiple image capturing angles of the working scene. The fixed viewing angle of the working scene provided in this embodiment includes at least one of the following viewing angles: a front viewing angle of the working scene, a top viewing angle of the working scene, a left viewing angle of the working scene, and a right viewing angle of the working scene.

[0168] The working scene under the formal perspective is the front view image of the working scene, the working scene under the overhead perspective is the overhead image of the working scene, the working scene under the left-side perspective is the left-side view image of the working scene, and the working scene under the right-side perspective is the right-side view image of the working scene.

[0169] The omnidirectional perspective of the work scene can be understood as the 360-degree image capture angle of the work scene. The work scene under the omnidirectional perspective is a 360-degree image of the work scene, that is, the image of the work scene under any perspective can be viewed through the omnidirectional perspective.

[0170] In an optional implementation of this embodiment, sending a mining request to the excavator includes sending a mining request including the target perspective selection information to the excavator. After receiving the mining request, the excavator acquires the operating scene point cloud data corresponding to the target perspective. The method for acquiring the operating scene point cloud data has been described in detail in the first embodiment of this application and will not be repeated here.

[0171] In another optional implementation of this embodiment, before the step of sending the mining request including the target perspective selection information to the excavator side, the method also includes the step of generating the mining request, that is, in response to the remote operator's selection operation of the target perspective, generating the mining request including the target perspective selection information.

[0172] The selection operation can be understood as a triggering operation of the remote operator on the target perspective, for example, triggering the selection of the target perspective through voice, or triggering the selection of the target perspective by clicking a selection control corresponding to the target perspective.

[0173] This embodiment provides an optional implementation method for selecting a target perspective, that is, through a first click operation of the remote operator on a first perspective selection control on the display screen, a first perspective corresponding to the first perspective selection control is selected as the target perspective.

[0174] Figure 7 It is a schematic diagram of the method for selecting a target viewing angle provided in this embodiment.

[0175] like Figure 7 As shown, a first perspective selection control 701 and a second perspective selection control 702 to be selected are displayed on the display screen of the remote control end. In response to the remote operator's click operation on the first perspective selection control 701, the first perspective corresponding to the first perspective selection control 701 is selected as the target perspective, and an excavation request including the first perspective selection information is sent to the excavator end, thereby enabling the excavator end to obtain point cloud data of the working scene under the first perspective.

[0176] For example: the first perspective corresponding to the first perspective selection control 701 is a fixed perspective, and the second perspective corresponding to the second perspective selection control 702 is an omnidirectional perspective. Then, in response to the remote operator's selection operation on the first perspective selection control 701, the fixed perspective is selected as the target perspective, and the mining request including the fixed perspective selection information is sent to the excavator end. The excavator end will obtain point cloud data including the front view perspective, top view perspective, left view perspective, and right view perspective of the working scene, and send the point cloud data including the front view perspective, top view perspective, left view perspective, and right view perspective of the working scene to the remote control end. The remote control end can obtain the point cloud model including the front view perspective, top view perspective, left view perspective, and right view perspective of the working scene based on the point cloud data, and the remote operator can see the front view 3D image, top view 3D image, left view 3D image, and right view 3D image of the working scene on the display screen of the remote control end. Of course, the front view 3D image, the top view 3D image, the left view 3D image, and the right view 3D image can be displayed on the display screen of the remote control end at the same time. The remote operator can also further select the fixed view angle to determine the fixed view angle displayed on the display screen. The remote operator can also rotate the point cloud model to display the front view 3D image, the top view 3D image, the left view 3D image, and the right view 3D image on the display screen respectively.

[0177] The excavation request provided in this embodiment may further include target perspective switching information, which may be understood as the image capture angle that the remote operator wishes to switch to during remote operation of the excavator. For example, the current target perspective is a fixed perspective, and the remote operator wishes to see images of the working scene from other perspectives except for the front 3D image, top 3D image, left 3D image, and right 3D image of the working scene during the operation. In this case, the target perspective may be switched, for example, the fixed perspective may be switched to an all-round perspective.

[0178] In an optional implementation of this embodiment, sending a mining request to the excavator includes sending a mining request including the target perspective switching information to the excavator. After receiving the mining request, the excavator acquires the operating scene point cloud data corresponding to the switched target perspective. The method for acquiring the operating scene point cloud data has been described in detail in the first embodiment of this application and will not be repeated here.

[0179] In another optional implementation of this embodiment, before the step of sending the mining request including the target perspective switching information to the excavator side, the method also includes the step of generating the mining request, that is, in response to the remote operator's switching operation on the target perspective, generating the mining request including the target perspective switching information.

[0180] The switching operation can be understood as a triggering operation of the remote operator on the target perspective to be switched, for example, triggering the switching of the target perspective by voice, or triggering the switching of the target perspective by clicking the selection control corresponding to the target perspective to be switched.

[0181] This embodiment provides an optional implementation method for switching the target perspective, that is, through a second click operation of the remote operator on the second perspective selection control on the display screen, the target perspective is switched from the first perspective to the second perspective corresponding to the second perspective selection control.

[0182] Figure 8 It is a schematic diagram of the method for switching the target viewing angle provided in this embodiment.

[0183] like Figure 8 As shown, a first perspective selection control 801 and a second perspective selection control 802 to be selected are displayed on the display screen of the remote control end. In response to the remote operator's click operation on the first perspective selection control 801, the first perspective corresponding to the first perspective selection control 801 is selected as the target perspective, and an excavation request including the first perspective selection information is sent to the excavator end, thereby enabling the excavator end to obtain point cloud data of the working scene under the first perspective.

[0184] Furthermore, in response to the remote operator's click operation on the second perspective selection control 802, the target perspective is switched from the first perspective to the second perspective corresponding to the second perspective selection control 802, and the mining request including the second perspective switching information is sent to the excavator side, so that the excavator side obtains the point cloud data of the working scene under the second perspective.

[0185] For example, if the first perspective selection control 801 corresponds to a fixed perspective, and the second perspective selection control 802 corresponds to an omnidirectional perspective, then in response to the remote operator's selection operation on the first perspective selection control 801, the fixed perspective is selected as the target perspective, and the excavation request including the fixed perspective selection information is sent to the excavator. The excavator will obtain point cloud data including the front view, top view, left view, and right view of the operating scene, and send the point cloud data including the front view, top view, left view, and right view of the operating scene to the remote control end, and the remote control end obtains the point cloud model including the front view, top view, left view, and right view of the operating scene based on the point cloud data. The remote operator can see the front view 3D image, top view 3D image, left view 3D image, and right view 3D image of the operating scene on the display screen of the remote control end.

[0186] Furthermore, in response to the remote operator clicking the second perspective selection control 802, the target perspective is switched from a fixed perspective to an omnidirectional perspective, and an excavation request including omnidirectional perspective selection information is sent to the excavator. The excavator acquires point cloud data including a 360-degree image capture perspective of the operating scene, and sends the point cloud data including a 360-degree image capture perspective of the operating scene to the remote control end, which acquires a point cloud model including a 360-degree image capture perspective of the operating scene based on the point cloud data. The remote operator can see the 360-degree image of the operating scene on the display screen by arbitrarily rotating the point cloud model on the screen of the remote control end.

[0187] Step S602: receiving the operation scene point cloud data sent by the excavator, constructing an operation scene point cloud model according to the operation scene point cloud data, and displaying the operation scene point cloud model on the display screen.

[0188] When the remote control terminal receives the point cloud data of the operation scene, it can use the point cloud rendering technology to construct a point cloud model of the operation scene. For example, the received point cloud data of the operation scene is rendered into a point cloud model of the operation scene through WebGL, and the point cloud model of the operation scene is presented on the screen of the remote control terminal.

[0189] The work scene point cloud data already includes the pixel coordinate information and depth information corresponding to each point in the work scene. By rendering the pixel coordinate information and depth information corresponding to each point included in the point cloud data on the remote control end, the corresponding point cloud model can be obtained.

[0190] Step S603: sending an excavation instruction to the excavator based on the point cloud model of the operation scene, so that the excavator controls the excavator to perform a target excavation action according to the excavation instruction.

[0191] After the point cloud model of the operation scene is displayed on the display screen of the remote control terminal, the remote operator can accurately see the current status of the operation scene through the point cloud model of the operation scene displayed on the display screen, such as the coordinate information and depth information of each point in the operation scene. Therefore, the remote operator can accurately send excavation instructions to the excavator through the remote control terminal based on the point cloud model of the operation scene.

[0192] The excavation instruction may be understood as a command sent by a remote operator to an excavator through a remote control terminal to control the excavator to perform a target excavation action, which may include pixel coordinate information and depth information corresponding to the target excavation point.

[0193] The target excavation point can be understood as the expected excavation point selected by the remote operator based on the point cloud model of the operation scene. Through the remote operator's selection of the target excavation point, the excavator can obtain the pixel coordinate information and depth information corresponding to the target excavation point, and then control the excavator to excavate the target excavation point according to the pixel coordinate information and depth information corresponding to the target excavation point.

[0194] In an optional implementation of this embodiment, sending an excavation instruction to the excavator includes sending an excavation instruction including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point to the excavator. After receiving the excavation instruction, the excavator will be controlled to excavate the target excavation point. The control method for controlling the excavator to excavate the target excavation point has been described in detail in the first embodiment of this application, and will not be repeated here.

[0195] In another optional implementation of the present embodiment, before the step of sending an excavation instruction including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point to the excavator side, the method also includes the step of generating the excavation instruction, that is, in response to the remote operator's selection operation of the target excavation point on the work scene point cloud model, generating the excavation instruction including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point.

[0196] The selection operation may be understood as a triggering operation of the remote operator on the target excavation point, for example, by clicking on the target excavation point on the point cloud model of the operation scene to trigger the selection of the target excavation point.

[0197] This embodiment provides an optional implementation method for selecting a target excavation point, that is, the target excavation point is selected by a remote operator performing a third click operation on the target excavation point on the work scene point cloud model.

[0198] Fig. 9 It is a schematic diagram of the method for selecting a target excavation point provided in this embodiment.

[0199] like Fig. 9 As shown, a point cloud model of the working scene is displayed on the display screen of the remote control end. In response to the remote operator's click operation on point 901 on the working scene point cloud model, point 901 is selected as the target excavation point, and an excavation instruction including pixel coordinate information and depth information corresponding to point 901 is sent to the excavator end, thereby enabling the excavator end to control the excavator to excavate point 901.

[0200] In combination with the first embodiment and the second embodiment, this embodiment further provides an optional implementation of a method for remotely controlling an excavator. Fig.10 It is a signaling flow chart of the method for remotely controlling an excavator provided in this embodiment.

[0201] like Fig.10 As shown, the method for remotely controlling an excavator provided in this embodiment includes the following steps:

[0202] Step S1001, the remote operator sends an excavation request to the excavator via the remote control terminal;

[0203] Step S1002, in response to the excavator receiving the excavation request, the excavator acquires the operation scene point cloud data according to the excavation request;

[0204] Step S1003, the excavator sends the operation scene point cloud data to the remote control end;

[0205] Step S1004, in response to the remote control terminal receiving the operation scene point cloud data, the remote control terminal constructs and displays the operation scene point cloud model according to the operation scene point cloud data;

[0206] Step S1005, the remote operator sends an excavation instruction to the excavator through the remote control terminal based on the point cloud model of the operation scene;

[0207] Step S1006, in response to the excavator receiving the excavation instruction, the excavator controls the excavator to perform a target excavation action according to the excavation instruction.

[0208] The above-mentioned second embodiment takes the remote control end as the main body and describes in detail the method for remotely controlling an excavator provided by the present application. The method for remotely controlling an excavator provided by the present application includes but is not limited to the method described in the second embodiment of the present application.

[0209] The third embodiment of the present application provides an excavator remote control system, Fig.11 It is a structural schematic diagram of the excavator remote control system provided in this embodiment.

[0210] like Fig.11 As shown, the excavator remote control system provided in this embodiment includes: a remote control terminal 1001 and an excavator terminal 1102;

[0211] The remote control end 1101 is used to send a mining request to the excavator end 1102 so that the excavator end 1102 obtains the work scene point cloud data according to the mining request; receives the work scene point cloud data sent by the excavator end 1102, constructs a work scene point cloud model according to the work scene point cloud data, and displays the work scene point cloud model on a display screen; based on the work scene point cloud model, sends a mining instruction to the excavator end 1102, so that the excavator end 1102 controls the excavator to perform the target mining action according to the mining instruction.

[0212] In an optional implementation of this embodiment, the mining request includes target perspective selection information, and sending the mining request to the excavator end 1102 includes: sending the mining request including the target perspective selection information to the excavator end 1102.

[0213] In another optional implementation of this embodiment, the remote control terminal 1101 is further configured to generate the mining request including the target perspective selection information in response to the remote operator's selection operation on the target perspective.

[0214] In an optional implementation of this embodiment, the remote operator's selection operation on the target perspective includes: the remote operator performs a first click operation on a first perspective selection control on the display screen to select a first perspective corresponding to the first perspective selection control as the target perspective.

[0215] In another optional implementation of this embodiment, the mining request further includes target perspective switching information, and sending the mining request to the excavator end 1102 includes: sending the mining request including the target perspective switching information to the excavator end 1102.

[0216] In an optional implementation of this embodiment, the remote control terminal 1101 is further used to generate the mining request including the target perspective switching information in response to the remote operator's switching operation on the target perspective.

[0217] In another optional implementation of this embodiment, the remote operator's switching operation on the target perspective includes: the remote operator performs a second click operation on the second perspective selection control on the display screen to switch the target perspective from the first perspective to a second perspective corresponding to the second perspective selection control.

[0218] In an optional implementation of this embodiment, the excavation instruction includes pixel coordinate information and depth information corresponding to the target excavation point, and sending the excavation instruction to the excavator end 1102 includes: sending a excavation instruction to the excavator end 1102 including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point.

[0219] In another optional implementation of the present embodiment, the remote control terminal 1101 is also used to generate a mining instruction including the pixel coordinate information corresponding to the target mining point and the depth information corresponding to the target mining point in response to the remote operator's selection operation on the target mining point on the work scene point cloud model.

[0220] In an optional implementation of the present embodiment, the selection operation of the remote operator on the target excavation point on the work scene point cloud model includes: a third click operation of the remote operator on the target excavation point on the work scene point cloud model.

[0221] The excavator end 1102 is used to receive the excavation request sent by the remote control end 1101, and obtain the work scene point cloud data according to the excavation request; send the work scene point cloud data to the remote control end 1101 so that the remote control end 1101 can build the work scene point cloud model according to the work scene point cloud data; receive the excavation instruction sent by the remote control end 1101 based on the work scene point cloud model, and control the excavator to perform the target excavation action according to the excavation instruction.

[0222] In an optional implementation of this embodiment, the mining request includes target perspective selection information; the target perspective includes a fixed perspective of the operation scene and an omnidirectional perspective of the operation scene; the fixed perspective of the operation scene includes at least one of the following perspectives:

[0223] The front viewing angle of the working scene, the top viewing angle of the working scene, the left viewing angle of the working scene, and the right viewing angle of the working scene.

[0224] In another optional implementation of this embodiment, the acquiring of the work scene point cloud data according to the mining request includes: acquiring the work scene point cloud data under the target perspective according to the mining request including the target perspective selection information.

[0225] In an optional implementation of this embodiment, sending the operation scene point cloud data to the remote control terminal 1101 so that the remote control terminal 1101 constructs an operation scene point cloud model according to the operation scene point cloud data includes:

[0226] The point cloud data of the operation scene under the target perspective is sent to the remote control terminal 1101, so that the remote control terminal 1101 can construct a point cloud model of the operation scene under the target perspective according to the point cloud data of the operation scene under the target perspective.

[0227] In another optional implementation of this embodiment, the operation scene point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene; the remote control terminal 1101 constructs an operation scene point cloud model according to the operation scene point cloud data, including:

[0228] The remote control terminal 1101 constructs the work scene point cloud model according to the pixel coordinate information and depth information corresponding to each point in the work scene, wherein the work scene point cloud model includes the pixel coordinate information corresponding to each point in the work scene and the depth information corresponding to each point in the work scene.

[0229] In an optional implementation of this embodiment, the excavation instruction includes pixel coordinate information and depth information corresponding to the target excavation point, and controlling the excavator to perform the target excavation action according to the excavation instruction includes:

[0230] According to the pixel coordinate information and depth information corresponding to the target excavation point, the excavator is controlled to excavate the target excavation point.

[0231] In another optional implementation of this embodiment, the excavator end 1102 is further used to obtain the current posture state parameters of the excavator in response to receiving the excavation instruction sent by the remote control end 1101 based on the work scene point cloud model, and the current posture state parameters include pixel coordinate information corresponding to the current point of the excavator and the current posture information of the excavator;

[0232] Calculating target posture state parameters of the excavator according to the pixel coordinate information and depth information corresponding to the target excavation point, wherein the target posture state parameters include the pixel coordinate information corresponding to the target excavation point and the target posture information of the excavator;

[0233] The motion trajectory of the excavator is calculated according to the current posture state parameters and the target posture state parameters, and the motion trajectory includes a point movement trajectory of the excavator and a posture change trajectory of the excavator.

[0234] In an optional implementation of this embodiment, controlling the excavator to excavate the target excavation point includes:

[0235] The excavator is controlled to move to the target excavation point according to the point movement trajectory, and the posture of the excavator is changed according to the posture change trajectory to excavate the target excavation point.

[0236] A fourth embodiment of the present application provides an excavator, Fig.12 It is a schematic diagram of the structure of the excavator provided in this embodiment.

[0237] like Fig.12 As shown, the excavator provided in this embodiment is equipped with a vehicle-mounted system, and the vehicle-mounted system at least includes: a first receiving component 1202, a data acquisition component 1202, a data sending component 1203, a second receiving component 1204, and a control component 1205.

[0238] The first receiving component 1201 is used to receive a mining request sent by a remote control terminal.

[0239] In an optional implementation of this embodiment, the mining request includes target perspective selection information; the target perspective includes a fixed perspective of the operation scene and an omnidirectional perspective of the operation scene; the fixed perspective of the operation scene includes at least one of the following perspectives:

[0240] The front viewing angle of the working scene, the top viewing angle of the working scene, the left viewing angle of the working scene, and the right viewing angle of the working scene.

[0241] The data acquisition component 1202 is used to acquire the operation scene point cloud data according to the mining request.

[0242] In another optional implementation of this embodiment, the acquiring of the work scene point cloud data according to the mining request includes: acquiring the work scene point cloud data under the target perspective according to the mining request including the target perspective selection information.

[0243] In an optional implementation of this embodiment, the operation scene point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene;

[0244] In another optional implementation of this embodiment, the data acquisition component 1202 includes: an image acquisition unit and a data calculation unit;

[0245] The image acquisition unit is used to acquire the original image of the operation scene;

[0246] The data calculation unit is used to calculate the pixel coordinate information and depth information corresponding to each point in the work scene based on the original image of the work scene, and the pixel coordinate information and depth information corresponding to each point constitute the work scene point cloud data.

[0247] In an optional implementation of this embodiment, the image acquisition unit includes a binocular camera arranged at the front end of the body of the excavator.

[0248] The data sending component 1203 is used to send the operation scene point cloud data to the remote control end.

[0249] In an optional implementation of this embodiment, the sending the operation scene point cloud data to the remote control terminal includes:

[0250] The point cloud data of the operation scene under the target perspective is sent to the remote control terminal.

[0251] The second receiving component 1204 is used to receive the mining instruction sent by the remote control terminal based on the point cloud model of the operation scene.

[0252] In another optional implementation of this embodiment, the excavation instruction includes pixel coordinate information and depth information corresponding to the target excavation point.

[0253] The control component 1205 is used to control the excavator to perform a target excavation action according to the excavation instruction.

[0254] In an optional implementation of this embodiment, controlling the excavator to perform a target excavation action according to the excavation instruction includes:

[0255] According to the pixel coordinate information and depth information corresponding to the target excavation point, the excavator is controlled to excavate the target excavation point.

[0256] In another optional implementation of this embodiment, the control component 1205 is further configured to:

[0257] In response to receiving the excavation instruction sent by the remote control terminal based on the work scene point cloud model, obtaining current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator;

[0258] Calculating target posture state parameters of the excavator according to the pixel coordinate information and depth information corresponding to the target excavation point, wherein the target posture state parameters include the pixel coordinate information corresponding to the target excavation point and the target posture information of the excavator;

[0259] The motion trajectory of the excavator is calculated according to the current posture state parameters and the target posture state parameters, and the motion trajectory includes a point movement trajectory of the excavator and a posture change trajectory of the excavator.

[0260] In an optional implementation of this embodiment, controlling the excavator to excavate the target excavation point includes:

[0261] The excavator is controlled to move to the target excavation point according to the point movement trajectory, and the posture of the excavator is changed according to the posture change trajectory to excavate the target excavation point.

[0262] The fifth embodiment of the present application provides an excavator remote control device, which is applied to the excavator. Fig.13 It is a schematic diagram of the structure of the excavator remote control device provided in this embodiment.

[0263] like Fig.13 As shown, the excavator remote control device provided in this embodiment includes: a point cloud data acquisition unit 1301, a point cloud data sending unit 1302, and an excavation action control unit 1303.

[0264] The point cloud data acquisition unit 1301 is used to receive a mining request sent by a remote control terminal, and acquire the point cloud data of the operation scene according to the mining request.

[0265] Optionally, the mining request includes target perspective selection information; the target perspective includes a fixed perspective of the operation scene and an omnidirectional perspective of the operation scene; the fixed perspective of the operation scene includes at least one of the following perspectives:

[0266] The front viewing angle of the working scene, the top viewing angle of the working scene, the left viewing angle of the working scene, and the right viewing angle of the working scene.

[0267] Optionally, obtaining the work scene point cloud data according to the mining request includes: obtaining the work scene point cloud data under the target perspective according to the mining request including the target perspective selection information.

[0268] The point cloud data sending unit 1302 is used to send the operation scene point cloud data to the remote control end, so that the remote control end can construct an operation scene point cloud model according to the operation scene point cloud data.

[0269] Optionally, the sending the operation scene point cloud data to the remote control end so that the remote control end constructs an operation scene point cloud model according to the operation scene point cloud data, includes:

[0270] The point cloud data of the operation scene under the target perspective is sent to the remote control end, so that the remote control end can construct a point cloud model of the operation scene under the target perspective according to the point cloud data of the operation scene under the target perspective.

[0271] Optionally, the operation scene point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene; the remote control end constructs an operation scene point cloud model according to the operation scene point cloud data, including:

[0272] The remote control terminal constructs the work scene point cloud model based on the pixel coordinate information and depth information corresponding to each point in the work scene, wherein the work scene point cloud model includes the pixel coordinate information corresponding to each point in the work scene and the depth information corresponding to each point in the work scene.

[0273] The excavation action control unit 1303 is used to receive the excavation instruction sent by the remote control end based on the operation scene point cloud model, and control the excavator to perform the target excavation action according to the excavation instruction.

[0274] Optionally, the excavation instruction includes pixel coordinate information and depth information corresponding to the target excavation point, and controlling the excavator to perform the target excavation action according to the excavation instruction includes:

[0275] According to the pixel coordinate information and depth information corresponding to the target excavation point, the excavator is controlled to excavate the target excavation point.

[0276] Optionally, the mining action control unit 1303 is further configured to:

[0277] In response to receiving the excavation instruction sent by the remote control terminal based on the work scene point cloud model, obtaining current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator;

[0278] Calculating target posture state parameters of the excavator according to the pixel coordinate information and depth information corresponding to the target excavation point, wherein the target posture state parameters include the pixel coordinate information corresponding to the target excavation point and the target posture information of the excavator;

[0279] The motion trajectory of the excavator is calculated according to the current posture state parameters and the target posture state parameters, and the motion trajectory includes a point movement trajectory of the excavator and a posture change trajectory of the excavator.

[0280] Optionally, controlling the excavator to excavate the target excavation point includes:

[0281] The excavator is controlled to move to the target excavation point according to the point movement trajectory, and the posture of the excavator is changed according to the posture change trajectory to excavate the target excavation point.

[0282] A sixth embodiment of the present application provides a remote control device for an excavator, which is applied to a remote control terminal, and the remote control terminal at least includes a display screen. Fig.14 It is a schematic diagram of the structure of the excavator remote control device provided in this embodiment.

[0283] like Fig.14 As shown, the excavator remote control device provided in this embodiment includes: an excavation request sending unit 1401, a point cloud model building unit 1402, and an excavation instruction sending unit 1403;

[0284] The mining request sending unit 1401 is used to send a mining request to the excavator side, so that the excavator side obtains the operation scene point cloud data according to the mining request.

[0285] Optionally, the mining request includes target perspective selection information, and sending the mining request to the excavator side includes: sending the mining request including the target perspective selection information to the excavator side.

[0286] Optionally, the mining request sending unit 1401 is further configured to generate the mining request including the target perspective selection information in response to a selection operation of the target perspective by the remote operator.

[0287] Optionally, the remote operator's selection operation on the target perspective includes: the remote operator's first click operation on a first perspective selection control on the display screen to select a first perspective corresponding to the first perspective selection control as the target perspective.

[0288] Optionally, the mining request further includes target perspective switching information, and sending the mining request to the excavator side includes: sending the mining request including the target perspective switching information to the excavator side.

[0289] Optionally, the mining request sending unit 1401 is further configured to generate the mining request including the target perspective switching information in response to the remote operator's switching operation on the target perspective.

[0290] Optionally, the remote operator's switching operation on the target perspective includes: the remote operator performing a second click operation on a second perspective selection control on the display screen to switch the target perspective from the first perspective to a second perspective corresponding to the second perspective selection control.

[0291] The point cloud model construction unit 1402 is used to receive the work scene point cloud data sent by the excavator, construct a work scene point cloud model according to the work scene point cloud data, and display the work scene point cloud model on the display screen.

[0292] The excavation instruction sending unit 1403 is used to send an excavation instruction to the excavator side based on the operation scene point cloud model, so that the excavator side controls the excavator to perform a target excavation action according to the excavation instruction.

[0293] Optionally, the excavation instruction includes pixel coordinate information and depth information corresponding to the target excavation point, and sending the excavation instruction to the excavator end includes: sending a excavation instruction to the excavator end including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point.

[0294] Optionally, the excavation instruction sending unit 1403 is also used to generate an excavation instruction including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point in response to the remote operator's selection operation on the target excavation point on the work scene point cloud model.

[0295] Optionally, the remote operator's selection operation on the target excavation point on the work scene point cloud model includes: a third click operation of the remote operator on the target excavation point on the work scene point cloud model.

[0296] A seventh embodiment of the present application provides an electronic device, Fig.15 It is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0297] like Fig.15 As shown, the electronic device provided in this embodiment includes: a memory 1501 and a processor 1502;

[0298] The memory 1501 is used to store computer instructions for executing the method for remotely controlling an excavator;

[0299] The processor 1502 is configured to execute computer instructions stored in the memory 1501 to perform the following operations:

[0300] Receiving a mining request sent by a remote control terminal, and acquiring point cloud data of an operation scene according to the mining request;

[0301] Sending the operation scene point cloud data to the remote control terminal, so that the remote control terminal constructs an operation scene point cloud model according to the operation scene point cloud data;

[0302] Receive the excavation instruction sent by the remote control terminal based on the work scene point cloud model, and control the excavator to perform the target excavation action according to the excavation instruction.

[0303] Optionally, the mining request includes target perspective selection information; the target perspective includes a fixed perspective of the operation scene and an omnidirectional perspective of the operation scene; the fixed perspective of the operation scene includes at least one of the following perspectives:

[0304] The front viewing angle of the working scene, the top viewing angle of the working scene, the left viewing angle of the working scene, and the right viewing angle of the working scene.

[0305] Optionally, obtaining the work scene point cloud data according to the mining request includes: obtaining the work scene point cloud data under the target perspective according to the mining request including the target perspective selection information.

[0306] Optionally, the sending the operation scene point cloud data to the remote control end so that the remote control end constructs an operation scene point cloud model according to the operation scene point cloud data, includes:

[0307] The point cloud data of the operation scene under the target perspective is sent to the remote control end, so that the remote control end can construct a point cloud model of the operation scene under the target perspective according to the point cloud data of the operation scene under the target perspective.

[0308] Optionally, the operation scene point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene; the remote control end constructs an operation scene point cloud model according to the operation scene point cloud data, including:

[0309] The remote control terminal constructs the work scene point cloud model based on the pixel coordinate information and depth information corresponding to each point in the work scene, wherein the work scene point cloud model includes the pixel coordinate information corresponding to each point in the work scene and the depth information corresponding to each point in the work scene.

[0310] Optionally, the excavation instruction includes pixel coordinate information and depth information corresponding to the target excavation point, and controlling the excavator to perform the target excavation action according to the excavation instruction includes:

[0311] According to the pixel coordinate information and depth information corresponding to the target excavation point, the excavator is controlled to excavate the target excavation point.

[0312] Optionally, before the step of controlling the excavator to excavate the target excavation point, the method further includes:

[0313] In response to receiving the excavation instruction sent by the remote control terminal based on the work scene point cloud model, obtaining current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator;

[0314] Calculating target posture state parameters of the excavator according to the pixel coordinate information and depth information corresponding to the target excavation point, wherein the target posture state parameters include the pixel coordinate information corresponding to the target excavation point and the target posture information of the excavator;

[0315] The motion trajectory of the excavator is calculated according to the current posture state parameters and the target posture state parameters, and the motion trajectory includes a point movement trajectory of the excavator and a posture change trajectory of the excavator.

[0316] Optionally, controlling the excavator to excavate the target excavation point includes:

[0317] The excavator is controlled to move to the target excavation point according to the point movement trajectory, and the posture of the excavator is changed according to the posture change trajectory to excavate the target excavation point.

[0318] Or, do the following:

[0319] Sending an excavation request to the excavator, so that the excavator obtains the point cloud data of the operation scene according to the excavation request;

[0320] Receiving the operation scene point cloud data sent by the excavator, constructing an operation scene point cloud model according to the operation scene point cloud data, and displaying the operation scene point cloud model on the display screen;

[0321] Based on the point cloud model of the operation scene, an excavation instruction is sent to the excavator end, so that the excavator end controls the excavator to perform a target excavation action according to the excavation instruction.

[0322] Optionally, the mining request includes target perspective selection information, and sending the mining request to the excavator side includes: sending the mining request including the target perspective selection information to the excavator side.

[0323] Optionally, before the step of sending the mining request including the target perspective selection information to the excavator end, the method further includes: generating the mining request including the target perspective selection information in response to a remote operator's selection operation on the target perspective.

[0324] Optionally, the remote operator's selection operation on the target perspective includes: the remote operator's first click operation on a first perspective selection control on the display screen to select a first perspective corresponding to the first perspective selection control as the target perspective.

[0325] Optionally, the mining request further includes target perspective switching information, and sending the mining request to the excavator side includes: sending the mining request including the target perspective switching information to the excavator side.

[0326] Optionally, before the step of sending the mining request including the target perspective switching information to the excavator end, the method further includes: generating the mining request including the target perspective switching information in response to the remote operator's switching operation on the target perspective.

[0327] Optionally, the remote operator's switching operation on the target perspective includes: the remote operator performing a second click operation on a second perspective selection control on the display screen to switch the target perspective from the first perspective to a second perspective corresponding to the second perspective selection control.

[0328] Optionally, the excavation instruction includes pixel coordinate information and depth information corresponding to the target excavation point, and sending the excavation instruction to the excavator end includes: sending a excavation instruction to the excavator end including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point.

[0329] Optionally, before the step of sending an excavation instruction including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point to the excavator end, the method further includes: in response to a remote operator's selection operation on the target excavation point on the work scene point cloud model, generating an excavation instruction including the pixel coordinate information corresponding to the target excavation point and the depth information corresponding to the target excavation point.

[0330] Optionally, the remote operator's selection operation on the target excavation point on the work scene point cloud model includes: a third click operation of the remote operator on the target excavation point on the work scene point cloud model.

[0331] An eighth embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium includes computer instructions. When the computer instructions are executed by a processor, they are used to implement the methods described in the various embodiments of the present application.

[0332] It should be noted that the relational terms such as "first" and "second" in this article are only used to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "include", "have", "include" and "includes" and other similar forms are the same in meaning, and the end of any one or more items after any of the above words is open-ended, and any of the above nouns does not mean that the one or more items have been exhaustively listed or are limited to the one or more items listed.

[0333] As used herein, unless expressly stated otherwise, the term "or" includes all possible combinations, except those that are not feasible. For example, if it is expressed that a database may include A or B, then unless otherwise specifically stated or not feasible, it may include database A, or B, or A and B. As a second example, if it is expressed that a database may include A, B, or C, then unless otherwise specifically stated or not feasible, the database may include database A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0334] It is worth noting that the above embodiments can be implemented by hardware or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-mentioned computer-readable medium. When the software is executed by a processor, it can execute the above-mentioned disclosed method. The computing unit and other functional units described in the present disclosure can be implemented by hardware or software, or a combination of hardware and software. Those of ordinary skill in the art will also understand that the above-mentioned multiple modules / units can be combined into one module / unit, and each of the above-mentioned modules / units can be further divided into multiple sub-modules / sub-units.

[0335] In the above detailed description, the embodiments have been described with reference to many specific details, which may vary from implementation to implementation. Certain adaptations and modifications may be made to the embodiments. For those skilled in the art, other embodiments may be readily apparent from the specific embodiments disclosed in the present application. This specification and examples are for exemplary purposes only, and the true scope and essence of the present application are described by the claims. The sequence of steps shown in the diagrams is also for the purpose of explanation only and is not intended to be limited to any particular steps or sequence. Therefore, those skilled in the art will appreciate that these steps may be performed in different orders when implementing the same method.

[0336] In the drawings and detailed description of the present application, exemplary embodiments are disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are used, these terms are only general and descriptive, and not for the purpose of limitation.

Claims

1. A method for remotely controlling an excavator, characterized in that: The method is applied to an excavator, and comprises: Receive a mining request sent by a remote control terminal, and acquire point cloud data of the operation scene according to the mining request, wherein the point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene, and the pixel coordinate information and the depth information are obtained by converting the point coordinate system of each point in the original image of the operation scene and calculating the point depth; Sending the operation scene point cloud data to the remote control terminal, so that the remote control terminal constructs an operation scene point cloud model according to the operation scene point cloud data; receiving an excavation instruction sent by the remote control terminal based on the point cloud model of the operation scene, and controlling the excavator to perform a target excavation action according to the excavation instruction, wherein the excavation instruction includes pixel coordinate information and depth information corresponding to a target excavation point, and the target excavation point is determined by a remote operator through a click operation on the point cloud model of the operation scene; The step of controlling the excavator to perform a target excavation action according to the excavation instruction includes: Acquire current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator; Calculating target posture information of the excavator according to the depth information corresponding to the target excavation point and the current posture information of the excavator; Calculate the point movement trajectory of the excavator according to the pixel coordinate information corresponding to the current point of the excavator and the pixel coordinate information corresponding to the target excavation point, and calculate the posture change trajectory of the excavator according to the current posture information and the target posture information; Controlling the excavator to move to the target excavation point according to the point movement trajectory, and changing the posture of the excavator according to the posture change trajectory to excavate the target excavation point; The posture information includes the angular position information between the excavator boom and the cabin, the forearm and the boom, the bucket and the forearm, and the cabin and the crawler.

2. The method according to claim 1, characterized in that The mining request includes target perspective selection information; the target perspective includes a fixed perspective of the operation scene and an omnidirectional perspective of the operation scene; the fixed perspective of the operation scene includes at least one of the following perspectives: The front viewing angle of the working scene, the top viewing angle of the working scene, the left viewing angle of the working scene, and the right viewing angle of the working scene.

3. The method according to claim 2, characterized in that The acquiring of the operating scene point cloud data according to the mining request includes: acquiring the operating scene point cloud data under the target perspective according to the mining request including the target perspective selection information.

4. The method according to claim 3, characterized in that The sending of the operation scene point cloud data to the remote control end so that the remote control end constructs an operation scene point cloud model according to the operation scene point cloud data includes: The point cloud data of the operation scene under the target perspective is sent to the remote control end, so that the remote control end can construct a point cloud model of the operation scene under the target perspective according to the point cloud data of the operation scene under the target perspective.

5. The method according to claim 1, characterized in that The operation scene point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene; The remote control terminal constructs a point cloud model of the operation scene according to the point cloud data of the operation scene, including: The remote control terminal constructs the work scene point cloud model based on the pixel coordinate information and depth information corresponding to each point in the work scene, wherein the work scene point cloud model includes the pixel coordinate information corresponding to each point in the work scene and the depth information corresponding to each point in the work scene.

6. A method for remotely controlling an excavator, characterized in that: The method is applied to a remote control terminal, the remote control terminal at least comprising a display screen, and the method comprises: Sending an excavation request to an excavator end, so that the excavator end obtains point cloud data of the operation scene according to the excavation request, wherein the point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene, and the pixel coordinate information and the depth information are obtained by converting the point coordinate system of each point in the original image of the operation scene and calculating the point depth; Receiving the operation scene point cloud data sent by the excavator, constructing an operation scene point cloud model according to the operation scene point cloud data, and displaying the operation scene point cloud model on the display screen; Based on the operation scene point cloud model, an excavation instruction is sent to the excavator end, so that the excavator end controls the excavator to perform a target excavation action according to the excavation instruction, wherein the excavation instruction includes pixel coordinate information and depth information corresponding to a target excavation point, and the target excavation point is determined by a remote operator through a click operation on the operation scene point cloud model; The step of controlling the excavator to perform a target excavation action according to the excavation instruction includes: Acquire current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator; Calculating target posture information of the excavator according to the depth information corresponding to the target excavation point and the current posture information of the excavator; Calculate the point movement trajectory of the excavator according to the pixel coordinate information corresponding to the current point of the excavator and the pixel coordinate information corresponding to the target excavation point, and calculate the posture change trajectory of the excavator according to the current posture information and the target posture information; Controlling the excavator to move to the target excavation point according to the point movement trajectory, and changing the posture of the excavator according to the posture change trajectory to excavate the target excavation point; The posture information includes the angular position information between the excavator boom and the cabin, the forearm and the boom, the bucket and the forearm, and the cabin and the crawler.

7. The method according to claim 6, characterized in that The mining request includes target perspective selection information, and sending the mining request to the excavator side includes: sending the mining request including the target perspective selection information to the excavator side.

8. The method according to claim 7, characterized in that Before the step of sending the mining request including the target perspective selection information to the excavator end, the method further includes: generating the mining request including the target perspective selection information in response to a remote operator's selection operation on the target perspective.

9. The method according to claim 8, characterized in that The remote operator's selection operation on the target perspective includes: the remote operator's first click operation on a first perspective selection control on the display screen to select a first perspective corresponding to the first perspective selection control as the target perspective.

10. The method according to claim 9, characterized in that: The mining request further includes target perspective switching information, and sending the mining request to the excavator side includes: sending the mining request including the target perspective switching information to the excavator side.

11. The method according to claim 10, characterized in that Before the step of sending the mining request including the target perspective switching information to the excavator end, the method further includes: generating the mining request including the target perspective switching information in response to the remote operator's switching operation on the target perspective.

12. The method according to claim 11, characterized in that The remote operator's switching operation on the target perspective includes: the remote operator performing a second click operation on a second perspective selection control on the display screen to switch the target perspective from the first perspective to a second perspective corresponding to the second perspective selection control.

13. A remote control system for an excavator, characterized in that: The system comprises: a remote control end and an excavator end; The remote control end is used to send an excavation request to the excavator end, so that the excavator end obtains the point cloud data of the working scene according to the excavation request, the point cloud data includes pixel coordinate information and depth information corresponding to each point in the working scene, and the pixel coordinate information and the depth information are obtained by converting the point coordinate system of each point in the original image of the working scene and calculating the point depth; receiving the working scene point cloud data sent by the excavator end, constructing a working scene point cloud model according to the working scene point cloud data, and displaying the working scene point cloud model on a display screen; based on the working scene point cloud model, sending an excavation instruction to the excavator end, so that the excavator end controls the excavator to perform a target excavation action according to the excavation instruction; The excavator end is used to receive the excavation request sent by the remote control end, obtain the operation scene point cloud data according to the excavation request; send the operation scene point cloud data to the remote control end, so that the remote control end can build the operation scene point cloud model according to the operation scene point cloud data; receive the excavation instruction sent by the remote control end based on the operation scene point cloud model, and control the excavator to perform the target excavation action according to the excavation instruction; The excavation instruction includes pixel coordinate information and depth information corresponding to a target excavation point, and the target excavation point is determined by a remote operator through a click operation on the point cloud model of the operation scene; The step of controlling the excavator to perform a target excavation action according to the excavation instruction includes: Acquire current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator; Calculating target posture information of the excavator according to the depth information corresponding to the target excavation point and the current posture information of the excavator; Calculate the point movement trajectory of the excavator according to the pixel coordinate information corresponding to the current point of the excavator and the pixel coordinate information corresponding to the target excavation point, and calculate the posture change trajectory of the excavator according to the current posture information and the target posture information; Controlling the excavator to move to the target excavation point according to the point movement trajectory, and changing the posture of the excavator according to the posture change trajectory to excavate the target excavation point; The posture information includes the angular position information between the excavator boom and the cabin, the forearm and the boom, the bucket and the forearm, and the cabin and the crawler.

14. An excavator, characterized in that: The excavator is equipped with a vehicle-mounted system, which at least includes: a first receiving component, a data acquisition component, a data sending component, a second receiving component, and a control component; The first receiving component is used to receive a mining request sent by a remote control terminal; The data acquisition component is used to acquire the point cloud data of the operation scene according to the mining request, wherein the point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene, and the pixel coordinate information and the depth information are obtained by converting the point coordinate system of each point in the original image of the operation scene and calculating the point depth; The data sending component is used to send the operation scene point cloud data to the remote control terminal; The second receiving component is used to receive an excavation instruction sent by the remote control end based on the operation scene point cloud model, wherein the excavation instruction includes pixel coordinate information and depth information corresponding to a target excavation point, and the target excavation point is determined by a remote operator through a click operation on the operation scene point cloud model; The control component is used to control the excavator to perform a target excavation action according to the excavation instruction; The step of controlling the excavator to perform a target excavation action according to the excavation instruction includes: Acquire current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator; Calculating target posture information of the excavator according to the depth information corresponding to the target excavation point and the current posture information of the excavator; Calculate the point movement trajectory of the excavator according to the pixel coordinate information corresponding to the current point of the excavator and the pixel coordinate information corresponding to the target excavation point, and calculate the posture change trajectory of the excavator according to the current posture information and the target posture information; Controlling the excavator to move to the target excavation point according to the point movement trajectory, and changing the posture of the excavator according to the posture change trajectory to excavate the target excavation point; The posture information includes the angular position information between the excavator boom and the cabin, the forearm and the boom, the bucket and the forearm, and the cabin and the crawler.

15. The excavator according to claim 14, characterized in that The data acquisition component includes an image acquisition unit and a data calculation unit; The image acquisition unit is used to acquire the original image of the operation scene; The data calculation unit is used to calculate the pixel coordinate information and depth information corresponding to each point in the work scene based on the original image of the work scene, and the pixel coordinate information and depth information corresponding to each point constitute the work scene point cloud data.

16. The excavator according to claim 15, characterized in that The image acquisition unit includes a binocular camera arranged at the front end of the body of the excavator.

17. A remote control device for an excavator, characterized in that: The device is applied to an excavator, and comprises: a point cloud data acquisition unit, a point cloud data sending unit, and an excavation action control unit; The point cloud data acquisition unit is used to receive a mining request sent by a remote control terminal, and acquire point cloud data of the operation scene according to the mining request, wherein the point cloud data includes pixel coordinate information and depth information corresponding to each point in the operation scene, and the pixel coordinate information and the depth information are obtained by converting the point coordinate system of each point in the original image of the operation scene and calculating the point depth; The point cloud data sending unit is used to send the operation scene point cloud data to the remote control terminal, so that the remote control terminal can construct an operation scene point cloud model according to the operation scene point cloud data; The excavation action control unit is used to receive an excavation instruction sent by the remote control terminal based on the operation scene point cloud model, and control the excavator to perform a target excavation action according to the excavation instruction, wherein the excavation instruction includes pixel coordinate information and depth information corresponding to a target excavation point, and the target excavation point is determined by a remote operator through a click operation on the operation scene point cloud model; The step of controlling the excavator to perform a target excavation action according to the excavation instruction includes: Acquire current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator; Calculating target posture information of the excavator according to the depth information corresponding to the target excavation point and the current posture information of the excavator; Calculate the point movement trajectory of the excavator according to the pixel coordinate information corresponding to the current point of the excavator and the pixel coordinate information corresponding to the target excavation point, and calculate the posture change trajectory of the excavator according to the current posture information and the target posture information; Controlling the excavator to move to the target excavation point according to the point movement trajectory, and changing the posture of the excavator according to the posture change trajectory to excavate the target excavation point; The posture information includes the angular position information between the excavator boom and the cabin, the forearm and the boom, the bucket and the forearm, and the cabin and the crawler.

18. A remote control device for an excavator, characterized in that: The device is applied to a remote control terminal, the remote control terminal at least includes a display screen, and the device includes: a mining request sending unit, a point cloud model building unit, and a mining instruction sending unit; The mining request sending unit is used to send a mining request to the excavator end, so that the excavator end obtains the point cloud data of the working scene according to the mining request, and the point cloud data includes the pixel coordinate information and depth information corresponding to each point in the working scene, and the pixel coordinate information and the depth information are obtained by converting the point coordinate system of each point in the original image of the working scene and calculating the point depth; The point cloud model construction unit is used to receive the operation scene point cloud data sent by the excavator, construct an operation scene point cloud model according to the operation scene point cloud data, and display the operation scene point cloud model on the display screen; The excavation instruction sending unit is used to send an excavation instruction to the excavator end based on the operation scene point cloud model, so that the excavator end controls the excavator to perform a target excavation action according to the excavation instruction, and the excavation instruction includes pixel coordinate information and depth information corresponding to a target excavation point, and the target excavation point is determined by a remote operator through a click operation on the operation scene point cloud model; The step of controlling the excavator to perform a target excavation action according to the excavation instruction includes: Acquire current posture state parameters of the excavator, wherein the current posture state parameters include pixel coordinate information corresponding to the current position of the excavator and current posture information of the excavator; Calculating target posture information of the excavator according to the depth information corresponding to the target excavation point and the current posture information of the excavator; Calculate the point movement trajectory of the excavator according to the pixel coordinate information corresponding to the current point of the excavator and the pixel coordinate information corresponding to the target excavation point, and calculate the posture change trajectory of the excavator according to the current posture information and the target posture information; Controlling the excavator to move to the target excavation point according to the point movement trajectory, and changing the posture of the excavator according to the posture change trajectory to excavate the target excavation point; The posture information includes the angular position information between the excavator boom and the cabin, the forearm and the boom, the bucket and the forearm, and the cabin and the crawler.

19. An electronic device, characterized in that: include: Memory and processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions to implement the method according to any one of claims 1-5 or claims 6-12.

20. A computer-readable storage medium having one or more computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the method according to any one of claims 1 to 5 or claims 6 to 12 is performed.

Images