Image acquisition method and remote control system for remote control of excavator

Abstract

The present application discloses an image acquisition method for remote control of an excavator, an excavator remote control system, and an excavator, the method comprising: acquiring a forward-looking original image, the forward-looking original image comprising at least two first images synchronously acquired by at least two first image acquisition devices arranged at the front end of the excavator body; acquiring a surround-looking original image, the surround-looking original image comprising at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator; parallel smoothing and splicing processing of the forward-looking original image and the surround-looking original image to obtain a forward-looking spliced ​​image and a surround-looking spliced ​​image. The method realizes parallel processing of the forward-looking original image and the surround-looking original image, expands the scene information that the operator can rely on when remotely controlling the excavator operation, and thus improves the accuracy of the operator's remote control of the excavator.

Description

Technical Field

[0001] The present application relates to the technical field of remote control of excavators, and in particular to an image acquisition method, device, electronic device and readable storage medium for remote control of excavators. The present application also relates to an excavator remote control system and an excavator. Background Art

[0002] With the development of computer remote control technology, remote control of excavators has also become an important research topic. Remote control of excavators mainly displays the working scene image of the excavator on the screen of the remote control terminal, so that the operator can control the excavator operation based on the working scene image on the remote control terminal. Therefore, image acquisition is a crucial step in remote control of excavators. Existing image acquisition methods for remote control of excavators often only involve the acquisition of the front image of the excavator. Although the operator can also control the excavator operation based on the front image of the excavator, there is still a risk of the excavator colliding with obstacles during movement or rotation.

[0003] Therefore, the existing image acquisition method for remote control of an excavator has the problem that the scene information involved in the working scene image provided to the remote control terminal is limited, resulting in a problem of low remote control accuracy. Summary of the invention

[0004] The present application provides an image acquisition method, device, electronic device, readable storage medium, an excavator remote control system, and an excavator for remote control of an excavator, so as to solve the problem that the existing image acquisition method for remote control of an excavator has limited scene information involved in the working scene image provided to the remote control end, resulting in low remote control accuracy.

[0005] The present application embodiment provides an image acquisition method for remote control of an excavator, the method comprising:

[0006] Acquire a forward-view original image, wherein the forward-view original image includes at least two first images synchronously acquired by at least two first image acquisition devices disposed at a front end of a body of the excavator;

[0007] Acquire a surround original image, wherein the surround original image includes at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator;

[0008] The forward-view original image and the surround-view original image are parallelly and smoothly stitched together to obtain a forward-view stitched image stitched together by the at least two first images and a surround-view stitched image stitched together by the at least four second images.

[0009] Optionally, the method further includes: sending the forward view stitched image and the surround view stitched image to a remote control terminal of the excavator, so as to display the forward view stitched image and the surround view stitched image on a screen of the remote control terminal of the excavator, so that a remote operator of the excavator can remotely control the excavator according to the forward view stitched image and the surround view stitched image displayed on the screen.

[0010] Optionally, the method further comprises: marking a projection position of the bucket in the front-view stitched image;

[0011] The forward view stitched image and the surround view stitched image marked with the projection position of the bucket are sent to the remote control terminal of the excavator, so that the forward view stitched image and the surround view stitched image marked with the projection position of the bucket are displayed on the screen of the remote control terminal of the excavator.

[0012] Optionally, the method further comprises: marking a distance scale in the front-view stitched image;

[0013] The forward view stitched image and the surround view stitched image marked with the distance scale are sent to the remote control terminal of the excavator, so that the forward view stitched image and the surround view stitched image marked with the distance scale and the surround view stitched image are displayed on the screen of the remote control terminal of the excavator.

[0014] Optionally, marking the projection position of the bucket in the front-view stitched image includes:

[0015] Obtain a first conversion parameter for converting the bucket coordinate system into the base coordinate system, and a second conversion parameter for converting the bucket coordinate system into the camera coordinate system;

[0016] Determine a third conversion parameter for converting the camera coordinate system into a base coordinate system according to the first conversion parameter and the second conversion parameter;

[0017] Calculate the second coordinate of the projection point of the bucket in the camera coordinate system according to the third conversion parameter and the first coordinate of the bucket in the base coordinate system;

[0018] Determine, according to the second coordinate, a third coordinate of the projection point of the bucket in a pixel coordinate system;

[0019] According to the third coordinate, the projection position of the bucket is marked in the front-view stitched image.

[0020] Optionally, the parallel smooth stitching processing of the forward view original image and the surround view original image includes: using a graphics processor to parallel smooth stitching processing of the forward view original image and the surround view original image, and a pre-programmed graphics processing program is built on the graphics processor.

[0021] Optionally, the smooth stitching process includes: image projection and image fusion.

[0022] Optionally, the performing smoothing stitching processing on the forward-looking original image to obtain a forward-looking stitched image stitched by stitching the at least two first images includes:

[0023] Based on a first projection algorithm, projecting the at least two first images onto a graphic surface to form a front-view combined image;

[0024] Based on a smoothing algorithm, a smoothing fusion process is performed on the overlapping area of ​​the at least two first images projected in the forward-looking combined image to form the forward-looking stitched image.

[0025] Optionally, projecting the at least two first images onto a graphic surface based on a first projection algorithm to form a forward-looking combined image includes:

[0026] respectively calculating the camera parameters corresponding to the at least two first image acquisition devices;

[0027] According to the camera parameters, the at least two first images are projected onto a cylindrical surface to form the forward-looking combined image.

[0028] Optionally, the performing smooth stitching processing on the surround original image to obtain a surround stitched image stitched by stitching the at least four second images includes:

[0029] Based on a second projection algorithm, projecting the at least four second images onto a ground plane to form a surround view combined image;

[0030] Based on the smoothing algorithm, smoothing and fusion processing is performed on the two-by-two overlapping areas of the at least four second images projected in the surround view combined image to form the surround view stitched image.

[0031] Optionally, projecting the at least four second images onto a ground plane based on a second projection algorithm to form a surround view combined image includes:

[0032] respectively calculating the camera parameters corresponding to the at least four second image acquisition devices and the projection parameters corresponding to the at least four second images;

[0033] The at least four second images are projected onto a ground plane according to the camera parameters and the projection parameters to form a surround view combined image.

[0034] Optionally, the smoothing algorithm includes: a multi-band fusion algorithm.

[0035] The embodiment of the present application also provides a remote control system for an excavator, the system comprising at least: a vehicle-mounted system and a remote operating system;

[0036] The vehicle-mounted system is configured on an excavator and includes: a first image acquisition component, a second image acquisition component, and an image processing component;

[0037] The first image acquisition component includes at least two first image acquisition devices, and the first image acquisition devices are arranged at the front end of the body of the excavator; the first image acquisition component is used to acquire a forward-looking original image of the excavator, and the forward-looking original image includes at least two first images synchronously acquired by the at least two first image acquisition devices;

[0038] The second image acquisition component includes at least four second image acquisition devices, and the second image acquisition devices are arranged around the body of the excavator; the second image acquisition component is used to acquire a surround original image of the excavator, and the surround original image includes at least four second images synchronously acquired by the at least four second image acquisition devices;

[0039] The image processing component is used to perform smooth stitching processing on the forward-looking original image and the surround-looking original image in parallel to form a forward-looking stitched image stitched by the at least two first images and a surround-looking stitched image stitched by the at least four second images;

[0040] The remote operating system at least includes: an image display component and an excavator control component;

[0041] The image display component includes at least one image display device for displaying the forward-view stitched image and the surround-view stitched image;

[0042] The excavator control component is used to control the excavator operation according to the control instructions of the remote operator of the excavator; the control instructions are determined by the remote operator of the excavator based on the forward-looking stitched image and the surround-looking stitched image displayed on the image display component.

[0043] Optionally, the image processing component is also used to send the forward view stitched image and the surround view stitched image to the remote operating system, so as to display the forward view stitched image and the surround view stitched image on the image display component of the remote operating system, so that the remote operator of the excavator can issue the control instructions to the excavator control component according to the forward view stitched image and the surround view stitched image displayed on the image display component.

[0044] Optionally, the image processing component is further used to mark the projection position of the bucket in the forward-view stitched image.

[0045] Optionally, the image processing component is further used to mark a distance scale in the forward-looking stitched image.

[0046] Optionally, the image processing component includes: a graphics processor, on which a pre-programmed graphics processing program is constructed to achieve parallel smooth stitching processing of the forward-looking original image and the surround-looking original image.

[0047] The embodiment of the present application further provides an excavator, on which a vehicle-mounted system is configured, the vehicle-mounted system comprising at least: a first image acquisition component, a second image acquisition component, and an image processing component;

[0048] The first image acquisition component includes at least two first image acquisition devices arranged at the front end of the excavator body, which synchronously acquire at least two first images within the working range of the excavator as forward-looking original images;

[0049] The second image acquisition component includes at least four second image acquisition devices arranged around the body of the excavator, which acquire at least four second images around the excavator as surround original images;

[0050] The image processing component includes a graphics processor, on which a preprogrammed graphics processing program is built, which performs parallel smooth stitching processing on the forward-looking original image and the surround-looking original image to form a forward-looking stitched image stitched by the at least two first images and a surround-looking stitched image stitched by the at least four second images.

[0051] Optionally, the first image acquisition component includes two fisheye cameras arranged at the front end of the excavator body; the two fisheye cameras synchronously capture two first images as the forward-looking original images; and there is an overlapping area between the two first images captured by the two fisheye cameras.

[0052] Optionally, the second image acquisition component includes four fisheye cameras respectively arranged at the front end, rear end, left end and right end of the excavator; the four fisheye cameras synchronously capture four second images as the surround original images; and there is an overlapping area between the two second images captured by two adjacent fisheye cameras.

[0053] Optionally, the image processing component further includes a bucket projection position marking module for marking the projection position of the bucket in the front-view stitched image.

[0054] Optionally, the image processing component further includes a distance scale marking module for marking a distance scale in the front-view stitched image.

[0055] Optionally, the image processing component further includes an image sending module for sending the forward-view stitched image and the surround-view stitched image to a remote control terminal of the excavator.

[0056] The embodiment of the present application also provides an image acquisition device for remote control of an excavator, the device comprising: a forward-looking original image acquisition unit, a surround-looking original image acquisition unit, and a smoothing splicing processing unit;

[0057] The forward-view original image acquisition unit is used to acquire a forward-view original image, wherein the forward-view original image includes at least two first images synchronously acquired by at least two first image acquisition devices disposed at the front end of the vehicle body of the excavator;

[0058] The surround original image acquisition unit is used to acquire a surround original image, wherein the surround original image includes at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator;

[0059] The smoothing stitching processing unit is used to perform parallel smoothing stitching processing on the forward-looking original image and the surround-looking original image to obtain a forward-looking stitching image stitched by the at least two first images and a surround-looking stitching image stitched by the at least four second images.

[0060] The embodiment of the present application also provides an electronic device, including: a memory and a processor;

[0061] The memory is used to store one or more computer instructions;

[0062] The processor is used to execute the one or more computer instructions to implement the above method.

[0063] The embodiment of the present application also provides a readable storage medium on which one or more computer instructions are stored. When the instructions are executed by a processor, the above method is executed.

[0064] Compared with the prior art, the image acquisition method for remote control of an excavator provided in the present application includes: acquiring a forward-looking original image, the forward-looking original image includes at least two first images synchronously acquired by at least two first image acquisition devices arranged at the front end of the excavator body; acquiring a surround-looking original image, the surround-looking original image includes at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator; performing parallel smooth splicing processing on the forward-looking original image and the surround-looking original image to obtain a forward-looking spliced ​​image spliced ​​by the at least two first images and a surround-looking spliced ​​image spliced ​​by the at least four second images. The method can simultaneously acquire a forward-looking original image representing the working scene in front of the excavator and a surround-looking original image representing the working scene around the excavator, and perform parallel smooth splicing processing on the acquired forward-looking original image and surround-looking original image to obtain a forward-looking spliced ​​image and a surround-looking spliced ​​image accordingly, so that the operator can control the excavator operation at the remote control end based on the forward-looking spliced ​​image and the surround-looking spliced ​​image at the same time. The image acquisition method for remote control of an excavator provided by the present application realizes parallel processing of the forward-looking original image and the surround-looking original image, expands the scene information that the operator can rely on when remotely controlling the excavator, and enables the operator to see the working scene in front of the excavator and the working scene around the excavator at the same time on the remote control end, thereby improving the accuracy of the operator's remote control of the excavator. The problem that the scene information involved in the working scene image provided to the remote control end is limited, resulting in low remote control accuracy, which exists in the existing image acquisition method for remote control of an excavator, is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is an application system diagram of an image acquisition method for remote control of an excavator provided in an embodiment of the present application;

[0066] Figure 2 is a flow chart of an image acquisition method for remote control of an excavator provided in the first embodiment of the present application;

[0067] Figure 3 This is an example diagram of performing smooth stitching processing on images provided in the first embodiment of the present application;

[0068] Figure 4 This is a flow chart of performing smooth stitching processing on the forward-looking original image provided by the first embodiment of the present application;

[0069] Figure 5 is a schematic diagram of a front-view stitched image provided by the first embodiment of the present application;

[0070] Figure 6This is a flow chart of performing smooth stitching processing on the surround original image provided by the first embodiment of the present application;

[0071] Figure 7 is a schematic diagram of a surround stitching image provided by the first embodiment of the present application;

[0072] Figure 8 is a flow chart of a method for marking a bucket projection position provided in the first embodiment of the present application;

[0073] Fig. 9 is a schematic diagram of a front-view stitched image marked with a projection position of a bucket and a distance scale provided in the first embodiment of the present application;

[0074] Fig.10 is a structural schematic diagram of an excavator remote control system provided by a second embodiment of the present application;

[0075] Fig.11 is a structural schematic diagram of an excavator provided in a third embodiment of the present application;

[0076] Fig.12 is a structural schematic diagram of an image acquisition device for remote control of an excavator provided in a fourth embodiment of the present application;

[0077] Fig.13 It is a schematic diagram of the structure of an electronic device provided in the fifth embodiment of the present application. DETAILED DESCRIPTION

[0078] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0079] The professional terms involved in the embodiments of the present application are explained below to facilitate understanding.

[0080] Remote control refers to the process of remote operators controlling remote equipment through computer networks. For example, remote control of computers means that remote operators operate remote computers through computer networks, such as remote operators controlling remote computers to help remote users solve computer problems. Another example: remote control of excavators means that remote operators use the control components of the remote operation end to operate the excavator at the construction site through the computer network, such as remotely controlling the excavator for mining.

[0081] The forward-view image refers to an image representing the scene in front of the excavator body, including a forward-view original image and a forward-view stitched image. The forward-view original image refers to an image of the scene in front of the excavator body captured by an image acquisition device (such as a camera) arranged in front of the excavator body. The forward-view stitched image refers to a combined image obtained by stitching multiple synchronously captured forward-view original images through a preset stitching algorithm. Remote operators are often unable to accurately control the excavator operation through a single forward-view original image. Therefore, the forward-view original image includes scene images of various angles in front of the excavator body synchronously captured by multiple image acquisition devices. By stitching these scene images of various angles into a forward-view stitched image, more scene information is provided to the remote operator, thereby improving the accuracy of the remote operator's remote control of the excavator.

[0082] Surround view images refer to images representing the scene around the excavator body, including original surround view images and stitched surround view images. The original surround view images refer to images of the scene around the excavator body captured by an image capture device (e.g., a camera) disposed around the excavator body. The stitched surround view image refers to a combined image obtained by stitching multiple synchronously captured original surround view images using a preset stitching algorithm.

[0083] The world coordinate system is the absolute coordinate system of the system. Before the user coordinate system is established, the coordinates of all points on the image are determined by the origin of this coordinate system.

[0084] The camera coordinate system takes the optical center of the camera as its origin, the x-axis and y-axis are parallel to the x-axis and y-axis of the image, and the z-axis is the optical axis of the camera, which is perpendicular to the graphic plane.

[0085] The pixel coordinate system is a direct coordinate system uv established with the upper left corner of the image as the origin and with pixels as the unit. The horizontal coordinate u and vertical coordinate v of a pixel represent the column and row number of the pixel in the image array respectively.

[0086] Excavators are an indispensable piece of engineering equipment in engineering construction, and they play an irreplaceable role in the fields of construction, transportation, and mining. At present, most of the operations of excavators are still completed by operators controlling the excavators. The working environment of excavators is often harsh, with a lot of dust, and there is a risk of disasters such as landslides and mudslides. If operators are in this environment for a long time, it will not only seriously affect their health, but also pose a great threat to their personal safety.

[0087] With the rapid development of remote control technology, remote control of excavators has also received attention. Remote control of excavators is to display the working scene image of the excavator on the screen of the remote control terminal, so that the remote operator can control the excavator operation based on the working scene image at the remote control terminal. Therefore, image acquisition is a crucial step in excavator remote control technology. The more scene information the working scene image includes and the clearer the image is, the more accurately the remote operator can remotely control the excavator.

[0088] At present, one image acquisition method for remote control of excavators is to obtain the scene image in front of the excavator through a single camera set at the front end of the excavator body, and the remote operator remotely controls the excavator operation based on the image. Although this method provides the scene image in front of the excavator operation, the image is collected by a single camera. Since the excavator itself is large in size, even if the camera is a wide-angle camera, the field of view of the image captured by it is relatively narrow compared to the excavator. When the remote operator remotely controls the excavator based on the image, there will be many blind spots, and it is impossible to control the excavator to perform comprehensive operations.

[0089] Another image acquisition method for remote control of excavators is to obtain two scene images in front of the excavator through two cameras set at the front of the excavator body, and then splice the two scene images to synthesize a front scene image. The remote operator remotely controls the excavator operation based on this image. The front scene image obtained by splicing has a wider field of view. The remote operator can obtain more scene information in front of the excavator through this image and can control the excavator to perform more comprehensive operations. However, this method only provides the scene image in front of the excavator, and there is still a risk that the excavator will collide with obstacles, on-site workers, etc. during movement or rotation. Therefore, providing more excavator operation scene information is crucial to improving the accuracy and safety of excavator remote control.

[0090] The existing image acquisition method for remote control of excavators cannot process and acquire more working scene images in parallel. When the excavator is remotely operated, the image refresh frame rate needs to be greater than 25hz. The existing image acquisition method for remote control of excavators can meet this standard when only the front scene image is processed. If other scene images to be processed are added, the image refresh frame rate cannot meet this standard, resulting in the remote operator being unable to remotely control the excavator in real time. Therefore, the existing image acquisition method for remote control of excavators cannot provide the remote control terminal with images containing more scene information.

[0091] In addition, the images of excavator operation scenes provided by the existing image acquisition methods for remote control of excavators often have obvious splicing marks at the image splicing points, which will provide a poor visual sensory experience for the remote operator.

[0092] Therefore, the existing image acquisition method for remote control of excavators has the problem that the scene information involved in the working scene image provided to the remote control terminal is limited, resulting in low remote control accuracy. In addition, the existing image acquisition method for remote control of excavators also has the problem that the working scene image provided to the remote control terminal has obvious splicing marks, resulting in poor visual perception of the operator.

[0093] In view of this, the present application provides an image acquisition method for remote control of an excavator, which realizes parallel smooth stitching processing of the forward-looking original image and the surround-looking original image, expands the scene information that the operator can rely on when remotely controlling the excavator, and enables the operator to see the working scene in front of the excavator and the working scene around the excavator at the same time at the remote control end, thereby improving the accuracy of the operator's remote control of the excavator.

[0094] The image acquisition method, device, electronic device, readable storage medium, excavator remote control system, and excavator described in the present application are further described in detail below in conjunction with specific embodiments and drawings.

[0095] Figure 1 This is an application system diagram of an image acquisition method for remote control of an excavator provided in an embodiment of the present application.

[0096] like Figure 1 As shown, the application system includes: an excavator end 101 and a remote control end 102. The excavator end 101 is communicatively connected with the remote control end 102 via a network. The excavator end 101 is located in an engineering construction scene, and includes at least an excavator and a vehicle-mounted system configured on the excavator. The vehicle-mounted system is deployed with the image acquisition method for remote control of an excavator provided in the present application, and provides the remote control end 102 with an image of the excavator's working scene. The remote control end 102 is located outside the engineering construction scene, and includes at least an image display component and an excavator control component. The image display component displays the working scene image of the excavator provided by the excavator end 101 in real time. The remote operator controls the excavator to operate in the engineering construction scene through the excavator control component based on the working scene image of the excavator displayed on the image display component.

[0097] The first embodiment of the present application provides an image acquisition method for remote control of an excavator. The method can be deployed on the excavator side, of course, it can also be deployed on the remote control side, and an image processing side between the excavator side and the remote control side can also be deployed. This embodiment takes the method deployed on the excavator side as an example for detailed description. The excavator side may include a vehicle-mounted system configured on the excavator, and the vehicle-mounted system may include an image acquisition component and an image processing component. The image acquisition component is used to acquire images of the excavator's working scene, and the image processing component is used to process the images of the excavator's working scene acquired by the image acquisition component, such as distortion calibration processing, stitching processing, etc. Therefore, the image acquisition method for remote control of an excavator provided in this embodiment and deployed on the excavator side is actually deployed on the image processing component in the vehicle-mounted system on the excavator side.

[0098] Figure 2 It is a flow chart of the image acquisition method for remote control of an excavator provided in this embodiment.

[0099] like Figure 2 As shown, the image acquisition method for remote control of an excavator provided in this embodiment includes the following steps:

[0100] Step S201, acquiring a forward-view original image, wherein the forward-view original image includes at least two first images synchronously acquired by at least two first image acquisition devices disposed at the front end of a vehicle body of the excavator.

[0101] The forward-view original image refers to an image of a scene in front of the excavator body collected by a first image acquisition device disposed at the front end of the excavator body. Since there are at least two first image acquisition devices disposed at the front end of the excavator body, the collected forward-view original image also includes at least two images, and the image included in the forward-view original image is defined as the first image.

[0102] The number of first images included in the forward-looking original image is the same as the number of first image acquisition devices arranged at the front end of the excavator body. For example, if three first image acquisition devices are arranged at the front end of the excavator body, the three first image acquisition devices will synchronously acquire images of the working scene in front of the excavator body, and the acquired forward-looking original image will include three first images.

[0103] The first image acquisition device refers to a device arranged at the front end of the excavator body and used to acquire images of the working scene in front of the excavator body.

[0104] After the first image acquisition device acquires images of the working scene in front of the excavator body, a forward-looking original image can be obtained. The first image acquisition device sends the forward-looking original image to the image processing component, and the image processing component obtains the forward-looking original image.

[0105] Of course, the operation process of the excavator is a continuous process. The operation scene in front of the excavator body captured by the first image acquisition device is actually a video, which is composed of a plurality of video frame images, each of which can be a forward-looking original image. In this embodiment, a video frame image in the video is used as a forward-looking original image to exemplarily illustrate the image acquisition method described in this embodiment.

[0106] This embodiment provides an optional implementation, that is, two fisheye cameras are set at the front end of the excavator body, specifically on the top of the cabin of the excavator body, so that the operating conditions of the upper arm, lower arm, and bucket of the excavator's excavation components can be collected while collecting the operating scene in front of the excavator. During the operation of the excavator, two first images are synchronously collected by the two fisheye cameras as forward-looking original images, and the collected forward-looking original images are sent to the image processing component, and the image processing component obtains the forward-looking original images. Since the fisheye camera is a panoramic camera that can achieve wide-range and no-dead-angle shooting, therefore, by setting two fisheye cameras at the front end of the excavator body, two first images with a larger field of view can be obtained, and the two first images will have overlapping areas for subsequent image stitching processing.

[0107] Step S202, acquiring a surround original image, wherein the surround original image includes at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator.

[0108] The surround original image refers to a scene image around the excavator body collected by a second image acquisition device arranged around the excavator body. Since there are at least four second image acquisition devices arranged around the excavator body, the collected surround original image also includes at least four images, and the image included in the surround original image is defined as the second image.

[0109] The number of second images included in the surround original image is the same as the number of second image acquisition devices arranged around the excavator body. For example, if eight second image acquisition devices are arranged around the excavator body, the eight second image acquisition devices will synchronously acquire images of the working scene around the excavator body, and the acquired surround original image will include eight second images.

[0110] The second image acquisition device refers to a device arranged around the excavator body and used to acquire images of the working scene around the excavator body.

[0111] After the second image acquisition device acquires images of the working scene around the excavator body, an original surround image can be obtained. The second image acquisition device sends the original surround image to the image processing component, and the image processing component obtains the original surround image.

[0112] Of course, the operation process of the excavator is a continuous process. The operation scene around the excavator body captured by the second image acquisition device is actually a video, which is composed of a number of video frame images, each of which can be a surround original image. In this embodiment, a video frame image in the video is used as the surround original image to exemplarily illustrate the image acquisition method described in this embodiment.

[0113] This embodiment provides an optional implementation method, that is, four fisheye cameras are respectively set at the front end, rear end, left end and right end of the excavator body, specifically above the crawler track of the excavator body, and the collected working scenes around the excavator include the scenes of the ground where the excavator is located. During the operation of the excavator, four second images are synchronously collected by the four fisheye cameras as surround original images, and the collected surround original images are sent to the image processing component, and the image processing component obtains the surround original images. Since the fisheye camera is a panoramic camera that can achieve wide-range and no-dead-angle shooting, four fisheye cameras are set around the body of the excavator to obtain four second images with a larger field of view, and there is an overlapping area between the two second images collected by two adjacent fisheye cameras for subsequent image stitching processing.

[0114] Step S203, performing parallel smoothing stitching processing on the forward-view original image and the surround-view original image to obtain a forward-view stitching image stitched by the at least two first images and a surround-view stitching image stitched by the at least four second images.

[0115] After the image processing component obtains the forward-view original image and the surround-view original image, it can process the forward-view original image and the surround-view original image in parallel, specifically, parallel smooth splicing the forward-view original image and the surround-view original image, smoothly splicing at least two first images into a forward-view spliced ​​image, and smoothly splicing at least four second images into a surround-view spliced ​​image. Parallel processing of the forward-view original image and the surround-view original image greatly shortens the image processing time and speeds up the image processing efficiency, making it possible to synchronously display the working scene in front of the excavator body and the working scene around the excavator body on the screen of the excavator remote control terminal.

[0116] This embodiment provides an optional implementation method for parallel processing of the forward-view original image and the surround-view original image, that is, a graphics processor is used to parallelly smoothly stitch the forward-view original image and the surround-view original image, and a pre-programmed graphics processing program is built on the graphics processor.

[0117] The Graphics Processing Unit (GPU), also known as display core, visual processor, display chip, is a microprocessor specifically used for image and graphics related computing. The GPU is a processor composed of many smaller and more specialized cores. When a processing task is divided and executed among multiple cores, these cores can provide powerful performance by working together. The GPU is a parallel processor, and the amount of tasks processed by the GPU per unit time is significantly greater than that of other processors. The parallel processing performance of the GPU is used to achieve parallel processing of the forward-looking original image and the surround-looking original image, which greatly accelerates the speed of processing the original image, and can make the refresh frame rate of the image in the excavator operation scene video displayed on the screen of the remote control end greater than 25hz.

[0118] The pre-programmed graphics processing program refers to a processing program formed by writing an algorithm for image smoothing and splicing processing in a programming model. Building the pre-programmed graphics processing program on the GPU can enable the GPU to implement computing tasks. For example: using cuda as a coding model, writing the smooth splicing algorithm in the cuda model as a pre-programmed graphics processing program, and building the program on the GPU, the GPU realizes parallel smooth splicing processing of the front-view original image and the surround original image. The cuda (Compute Unified Device Architecture) is a general parallel computing platform and programming model. Based on cuda programming, the parallel computing engine of the GPU can solve more complex computing problems more efficiently.

[0119] In other words, the GPU provides the conditions for parallel processing of the forward-looking original image and the surround-looking original image, and the CUDA that has written the smooth stitching algorithm provides a method for smoothly stitching the forward-looking original image and the surround-looking original image.

[0120] The smoothing splicing processing method provided in this embodiment includes: image projection and image fusion.

[0121] The image projection refers to projecting at least two first images in the acquired forward-looking original image or at least four second images in the surround-looking original image onto a fixed plane to form a forward-looking combined image or a surround-looking combined image, thereby realizing the combined splicing of images. In the image projection stage, although the transformation of multiple original images into one image has been realized, only the combined state of multiple original images is displayed in the image, and there are obvious splicing marks between the multiple original images.

[0122] The image fusion refers to fusing multiple original images displayed in the forward-looking combined image or the surround-looking combined image, specifically, fusing the images through the overlapping areas between the original images, so that there are no obvious splicing marks between the original images in the forward-looking combined image or the surround-looking combined image, thereby realizing the fusion and splicing of the images. In the image fusion stage, at least two first images acquired by at least two first image acquisition devices can be made to be like images acquired by one image acquisition device, and at least four second images acquired by at least four second image acquisition devices can be made to be like images acquired by one image acquisition device.

[0123] Figure 3 This is an example diagram of smooth stitching of images provided in this embodiment.

[0124] like Figure 3 As shown, Figure 301 and Figure 302 are two original images to be spliced. First, image projection processing is performed on Figure 301 and Figure 302 to form Figure 303. Figure 303 is a combined image of Figure 301 and Figure 302. There are obvious splicing marks in Figure 303, and it can be clearly seen that Figure 303 is an image composed of Figure 301 and Figure 302. Secondly, image fusion processing is performed on Figure 303 to form Figure 304. During the image fusion processing, the splicing marks in Figure 303 are removed, making Figure 304 more like an image taken at one time by an image acquisition device.

[0125] The smooth stitching processing of the forward-looking original image and the smooth stitching processing of the surround-looking original image are respectively described in detail below.

[0126] This embodiment provides an optional implementation method for performing smooth splicing processing on the forward-looking original image. Figure 4 This is a flowchart of the smooth stitching process of the forward-looking original image provided by this embodiment.

[0127] like Figure 4 As shown, the present embodiment provides a method for smoothing and splicing the front-view original image, including the following steps:

[0128] Step S401: Based on a first projection algorithm, project the at least two first images onto a graphic surface to form a forward-looking combined image.

[0129] The first projection algorithm refers to a projection algorithm for the forward-looking original image, specifically, projecting at least two first images onto a graphic surface, such as: projecting onto a plane, for example: projecting onto a cylindrical surface, for example: projecting onto a spherical surface.

[0130] Based on the first projection algorithm, projecting at least two first images onto a graphic surface to form a front-view combined image may include the following steps:

[0131] First, camera parameters corresponding to the at least two first image acquisition devices are calculated respectively.

[0132] The camera parameters include camera intrinsics and camera extrinsics, where camera extrinsics are parameters involved in converting an image from a world coordinate system to a camera coordinate system, such as rotation matrix parameters and translation vectors; camera intrinsics are parameters involved in converting an image from a camera coordinate system to a pixel coordinate system, such as focal length and optical center offset. Camera intrinsics are parameters determined when the camera leaves the factory, and camera extrinsics are related to the fixation of the camera. Camera parameters can be calculated by camera calibration.

[0133] The calculation of the camera parameters corresponding to the first image acquisition device described in this embodiment is actually the process of camera calibration, that is, the process of solving the camera parameters. Among them, the rotation matrix is ​​an important camera external parameter. The rotation matrix is ​​the parameter involved in converting the image from the world coordinate system to the camera coordinate system. After the image vector is multiplied by the rotation matrix, only the direction of the image vector is changed, and the size and chirality of the image vector are not changed.

[0134] Taking the example of setting two fisheye cameras at the front end of the excavator body, the camera calibration process is explained, including: feature extraction step, feature matching step, and camera parameter evaluation step.

[0135] The feature extraction step includes detecting feature points of the two acquired first images. Specifically, SIFT, SURF, ORB and other algorithms may be used to detect the feature points of the first images to obtain feature vectors of the feature points of the two first images.

[0136] The feature matching step includes matching the extracted feature points. Specifically, the nearest neighbor and next nearest neighbor algorithms can be used to compare the Euclidean distances between the feature vectors of the feature points of the two first images, so as to determine whether the two compared feature points can match. Finally, the confidence levels of the two best matches are saved.

[0137] The camera parameter evaluation step includes estimating the camera parameters of the two fisheye cameras according to the matching results of the feature points of the two first images. Specifically, the camera parameters can be roughly estimated by using the Homography Based Estimator algorithm, and then the camera parameters can be finely estimated by the beam averaging method.

[0138] Since the relative positions of the first image acquisition devices will not change after at least two first image acquisition devices are arranged at the front end of the excavator body, the camera parameters only need to be estimated once and can be used repeatedly.

[0139] Second, according to the camera parameters, the at least two first images are projected onto a cylindrical surface to form the forward-looking combined image.

[0140] The camera parameters of the first image acquisition device arranged at the front end of the excavator have been obtained in the previous step. The image acquired by the first image acquisition device can be projected according to the camera parameters. Specifically, at least two first images can be projected on a cylindrical surface to form a forward-looking combined image.

[0141] The projection of the image includes the first transformation from the world coordinate system to the camera coordinate system and the second transformation from the camera coordinate system to the pixel coordinate system. The first transformation is the transformation from a 3D image to a 3D image, and the main parameters involved are the camera extrinsic parameters. The second transformation is the transformation from a 3D image to a 2D image, and the main parameters involved are the camera intrinsic parameters. After obtaining the camera parameters through the camera calibration step, each projection of the forward-looking original image can be calculated using the camera parameters.

[0142] Step S402: Based on a smoothing algorithm, smoothing and fusing the overlapping area of ​​the at least two first images projected in the forward-looking combined image to form the forward-looking stitched image.

[0143] At least two first images are combined into a front-view combined image through step S401, but the overlapping area between the first images will have obvious stitching marks, which is caused by different blurring degrees and brightness of the multiple first images. Through the smoothing algorithm of this step, the overlapping area between the first images can be smoothly fused, thereby eliminating the stitching marks, and forming a front-view stitched image that is visually sensed as if it was collected by one image collection device.

[0144] The smoothing algorithm provided in this embodiment may be a multi-band fusion algorithm.

[0145] The multi-band blending (Multi-Band Blender), also known as Laplace pyramid fusion, is to construct Laplace pyramids for the images to be fused (the Laplace operator can extract the frequency information of the image. In the Laplace pyramid, the higher the image component is, the higher the frequency is). Then, the image components of the same layer (the image components of the same frequency band) are fused according to a certain rule; different layers of image components (image components of different frequency bands) are fused according to different rules, the high-frequency parts are fused slowly, and the low-frequency parts are fused quickly; finally, the image components of each frequency band are re-added to obtain the final fusion result.

[0146] Based on the smoothing algorithm, the specific steps of performing smooth fusion processing on the overlapping area of ​​at least two first images projected in the forward-looking combined image include:

[0147] First, respectively calculate the Laplacian pyramids of at least two first images projected in the front-view combined image;

[0148] Second, image components in the same frequency band are fused one by one;

[0149] Third, starting from the high frequency segment, each Laplacian pyramid is expanded in sequence until the resolution is the same as the corresponding first image;

[0150] Fourth, the image components of each frequency band are sequentially added to obtain a forward-looking stitched image.

[0151] Figure 5 is a schematic diagram of the front-view spliced ​​image provided by this embodiment. Figure 5 As shown, through the image projection and image fusion method provided by this embodiment, the horizontal field of view angle of the forward-looking stitched image obtained exceeds 200°, the vertical field of view angle exceeds 100°, and there is no stitching mark in the forward-looking stitched image, which is visually the same as an image captured by an image acquisition device.

[0152] This embodiment provides an optional implementation method for performing smooth splicing processing on surround view original images. Figure 6 This is a flowchart of the smooth stitching process of the surround original image provided by this embodiment.

[0153] like Figure 6 As shown, the present embodiment provides a method for smoothly stitching the surround original image, including the following steps:

[0154] Step S601: Based on a second projection algorithm, project the at least four second images onto a ground plane to form a surround view combined image.

[0155] The second projection algorithm refers to a projection algorithm for the surround original image, specifically, projecting at least four second images onto the ground plane.

[0156] Based on the second projection algorithm, projecting at least four second images onto the ground plane to form a surround view combined image may include the following steps:

[0157] First, camera parameters corresponding to the at least four second image acquisition devices and projection parameters corresponding to the at least four second images are calculated respectively.

[0158] Since the surround original image is projected onto the ground plane, the projection process involves not only camera parameters but also projection parameters, such as distortion correction parameters, projection transformation parameters, etc. These parameters can be calculated by camera calibration methods.

[0159] Taking the example of setting a fisheye camera at the front, rear, left and right ends of the excavator body respectively, the camera calibration process is explained, including: camera intrinsic parameters and distortion correction parameter acquisition steps, projective transformation parameter acquisition steps, feature extraction steps, feature matching steps and homography matrix evaluation steps.

[0160] The camera intrinsic parameters and distortion correction parameter acquisition steps are mainly performed by placing a chessboard calibration plate around the excavator, collecting a number of images with the chessboard calibration plate through four fisheye cameras, dedistorting and correcting the image according to the chessboard calibration plate in the image, obtaining the distortion correction parameters, and obtaining the camera intrinsic parameters at the same time.

[0161] The projective transformation parameter acquisition step is mainly to set horizontally placed QR code calibration plates at four diagonal corners of the excavator (it is necessary to make every two cameras able to see the same QR code calibration plate), collect four images with the QR code calibration plates through four fisheye cameras, and obtain the projective transformation parameters based on the matching of the QR code corner points between the images.

[0162] The feature extraction step includes converting the four second images into bird's-eye view images according to the distortion correction parameters and the projective transformation parameters, and performing feature point detection on the four bird's-eye view images.

[0163] The feature matching step includes matching the extracted feature points. Specifically, the feature points of the four bird's-eye view images may be roughly matched using the nearest neighbor and next nearest neighbor algorithms.

[0164] The homography matrix evaluation includes solving the homography matrix according to the rough matching result.

[0165] Since the relative positions of the second image acquisition devices will not change after at least four second image acquisition devices are arranged around the excavator body, the above parameters only need to be estimated once and can be used repeatedly.

[0166] Second, according to the camera parameters and the projection parameters, the at least four second images are projected onto a ground plane to form a surround view combined image.

[0167] The camera parameters of the second image acquisition device set around the excavator and the projection parameters corresponding to the second image have been obtained in the previous step. The image acquired by the second image acquisition device can be projected according to the camera parameters and the projection parameters. Specifically, at least four second images can be projected on the ground plane to form a surround combined image.

[0168] Step S602: Based on the smoothing algorithm, smoothing and fusing the overlapping areas of the at least four second images projected in the surround view combined image to form the surround view stitched image.

[0169] At least four second images are combined into a surround view combined image through step S601, but there will be obvious stitching marks in the overlapping areas of the second images, which is caused by different blur levels and brightness of the multiple second images. Through the smoothing algorithm of this step, the overlapping areas of the at least four second images can be smoothly fused, thereby eliminating the stitching marks and forming a surround view stitched image that looks like it is collected by one image collection device in terms of visual perception.

[0170] The smoothing algorithm provided in this embodiment may be a multi-band fusion algorithm.

[0171] The multi-band blending (Multi-Band Blender), also known as Laplace pyramid fusion, is to construct Laplace pyramids for the images to be fused (the Laplace operator can extract the frequency information of the image. In the Laplace pyramid, the higher the image component is, the higher the frequency is). Then, the image components of the same layer (the image components of the same frequency band) are fused according to a certain rule; different layers of image components (image components of different frequency bands) are fused according to different rules, the high-frequency parts are fused slowly, and the low-frequency parts are fused quickly; finally, the image components of each frequency band are re-added to obtain the final fusion result.

[0172] Based on the smoothing algorithm, the specific steps of performing smooth fusion processing on the two-by-two overlapping areas of the at least four second images projected in the surround view combined image include:

[0173] First, respectively calculate the Laplacian pyramids of at least four second images projected in the surround view combined image;

[0174] Second, image components in the same frequency band are fused one by one;

[0175] Third, starting from the high frequency segment, each Laplacian pyramid is expanded in sequence until the resolution is the same as the corresponding second image;

[0176] Fourth, the image components of each frequency band are sequentially added to obtain a surround stitching image.

[0177] Figure 7 is a schematic diagram of the surround stitching image provided by this embodiment. Figure 7 As shown, through the image projection and image fusion method provided by this embodiment, the surround stitching image obtained is visually the same as an image collected by an image acquisition device.

[0178] The above-mentioned smoothing and splicing processing of the forward-view original image and the smoothing and splicing processing of the surround-view original image are performed in parallel in the image processing component.

[0179] After completing the smooth stitching processing of the image, the method provided in this embodiment also includes: sending the forward view stitching image and the surround view stitching image to the remote control terminal of the excavator, so as to display the forward view stitching image and the surround view stitching image on the screen of the remote control terminal of the excavator, so that the remote operator of the excavator can remotely control the excavator according to the forward view stitching image and the surround view stitching image displayed on the screen.

[0180] An optional implementation method is: the image processing component sends the processed forward-view stitching image and surround-view stitching image to the remote control terminal of the excavator, and the front working scene video of the excavator composed of each frame of the forward-view stitching image and the surrounding working scene video of the excavator composed of each frame of the surround-view stitching image can be seen on the screen of the remote control terminal at the same time. The remote operator can control the excavator operation more safely and accurately based on the front working scene video and the surrounding working scene video displayed on the screen at the same time.

[0181] Since the stitched images are all two-dimensional images, which have no depth information, the remote operator can more accurately control the excavator to operate if he knows the position where the bucket can land on the ground. Therefore, the method provided in this embodiment also includes: marking the projection position of the bucket in the forward-looking stitched image; sending the forward-looking stitched image and the surround-looking stitched image marked with the projection position of the bucket to the remote control terminal of the excavator, so as to display the forward-looking stitched image and the surround-looking stitched image marked with the projection position of the bucket on the screen of the remote control terminal of the excavator.

[0182] The projection position of the bucket refers to the position of the bucket projected on the ground. By marking the projection position of the bucket in the front-view stitched image, the remote operator can know the position where the bucket can fall on the ground in the front-view stitched image.

[0183] This embodiment provides an optional implementation method for marking the projection position of the bucket in the front-view stitched image. Figure 8 It is a flow chart of the method for marking the projection position of the bucket provided in this embodiment.

[0184] like Figure 8 As shown, the method of marking the projection position of the bucket in the front-view stitched image provided in this embodiment includes the following steps:

[0185] Step S801, obtaining a first conversion parameter T_b2g for converting the bucket coordinate system gripper_link to the base coordinate system base_link, and a second conversion parameter T_c2g for converting the bucket coordinate system gripper_link to the camera coordinate system camera_link.

[0186] The coordinate system of the boom root of the excavator is used as the base coordinate system base_link. The method for obtaining the first conversion parameter T_b2g includes: installing posture sensors on the boom, the forearm, and the bucket to measure the three-dimensional motion posture of the excavator bucket during the movement process, thereby obtaining the conversion relationship T_b2g of the bucket coordinate system gripper_link relative to the base coordinate system base_link. The method for obtaining the second conversion parameter T_c2g includes: pasting a calibration plate at the root of the forearm, and using a camera calibration method to calculate the conversion relationship T_c2g of the bucket coordinate system gripper_link relative to the camera coordinate system camera_link.

[0187] Step S802: Determine a third conversion parameter T_b2c for converting the camera coordinate system camera_link into the base coordinate system base_link according to the first conversion parameter T_b2g and the second conversion parameter T_c2g.

[0188] The conversion relationship among the first conversion parameter T_b2g, the second conversion parameter T_c2g, and the third conversion parameter T_b2c is shown in the following formula:

[0189] T b2c =T b2g ×T g2c =T b2g ×inv(T c2g )

[0190] Step S803, calculating the second coordinate B_c of the projection point of the bucket in the camera coordinate system camera_link according to the third conversion parameter T_b2c and the first coordinate B_b of the bucket in the base coordinate system base_link.

[0191] Assume that the vertical distance from the base of the excavator's boom to the ground is H, and the coordinates of the bucket in the base coordinate system base_link are (x, y, z), that is, the first coordinate B_b = [x, y, z], then the coordinates of the bucket's projection point on the ground in the base coordinate system base_link are (x, y, -H), which is transformed into homogeneous coordinates (x, y, -H, 1), and B represents the homogeneous coordinates of the bucket's projection point in the base coordinate system base_link, that is, B = [x, y, -H, 1], then the second coordinate B_c of the projection point in the camera coordinate system camera_link can be calculated by the following formula:

[0192] B c =T c2b ×B=inv(T b2c )×B

[0193] Step S804: Determine the third coordinate of the projection point of the bucket in the pixel coordinate system according to the second coordinate B_c.

[0194] Through the above steps, the second coordinate of the projection point in the camera coordinate system camera_link is obtained, and the coordinate of the projection point in the pixel coordinate system, that is, the third coordinate, can be obtained based on the camera intrinsic parameters.

[0195] Step S805: marking the projection position of the bucket in the front-view stitched image according to the third coordinate.

[0196] The third coordinate represents the specific position of the projection point in the forward-looking original image. Therefore, the projection position can be marked in the forward-looking spliced ​​image according to the specific position of the projection point in the forward-looking original image.

[0197] After the projection position of the bucket is marked in the forward-view stitched image, the method provided in this embodiment further includes: marking a distance scale in the forward-view stitched image; sending the forward-view stitched image and the surround-view stitched image marked with the distance scale to the remote control terminal of the excavator, so as to display the forward-view stitched image and the surround-view stitched image marked with the distance scale on the screen of the remote control terminal of the excavator.

[0198] The method of marking the distance scale in the forward-looking stitched image is the same as the method of marking the bucket projection position. Marking the bucket projection position is actually to take the bucket projection position as a projection point and convert the basic coordinate system to the pixel coordinate system for the projection point. Marking the distance scale can also decompose the distance scale into several scale points. By converting each scale point from the basic coordinate system to the pixel coordinate system, several scale points are marked on the forward-looking stitched image, thereby obtaining a distance scale composed of several scale points.

[0199] Furthermore, the method provided in this embodiment also includes: sending the forward view stitched image marked with the projection position of the bucket and the distance scale, and the surround view stitched image to the remote control terminal of the excavator, so as to display the forward view stitched image marked with the projection position of the bucket and the distance scale, and the surround view stitched image on the screen of the remote control terminal of the excavator.

[0200] Fig. 9 Schematic diagram of a front-view stitched image marked with the projection position of the bucket and the distance scale provided in this embodiment.

[0201] like Fig. 9 As shown, the forward-looking stitched image marked with the bucket's projected position and distance scale is sent to the excavator's remote control terminal through the image processing component and displayed on the remote control terminal's screen. In addition to the working scene in front of the excavator body, the forward-looking stitched image also displays three arc-shaped marks 901 and an elliptical mark 902. The arc-shaped mark 901 is the distance scale, and the three distance scales represent the distances of 8 meters, 10 meters, and 12 meters from the center of the excavator, respectively. The elliptical mark 902 is the projected position of the bucket, that is, the position where the bucket falls on the ground. Fig. 9 It can be seen that the projection position of the bucket in the front-view stitched image is located on the 8-meter distance scale, which means that if the bucket is dropped to the ground at this time, it will fall 8 meters away from the center of the excavator. Of course, the distance from the distance scale marked in the front-view stitched image to the center of the excavator can be adjusted according to factors such as the size of the excavator and the scope of the working scene. For example: if you want to perform close-range excavation work in a scene with a narrow working range, you can mark distance scales of 3 meters, 5 meters, and 8 meters from the center of the excavator in the front-view stitched image. A distance scale that is too large will lose its meaning in this scene.

[0202] The above-mentioned first embodiment describes in detail the image acquisition method for remote control of an excavator described in this application in an optional implementation manner. The method described in this application includes but is not limited to the implementation manner described in the first embodiment of this application.

[0203] The second embodiment of the present application provides an excavator remote control system, Fig.10 It is a structural schematic diagram of the excavator remote control system provided in this embodiment.

[0204] like Fig.10 As shown, the excavator remote control system provided in this embodiment includes: a vehicle-mounted system 1001 and a remote operation system 1002.

[0205] The vehicle-mounted system 1001 is configured on an excavator and includes: a first image acquisition component 10011, a second image acquisition component 10012, and an image processing component 10013;

[0206] The first image acquisition component 10011 includes at least two first image acquisition devices, and the first image acquisition devices are arranged at the front end of the body of the excavator; the first image acquisition component 10011 is used to acquire the front view original image of the excavator, and the front view original image includes at least two first images synchronously acquired by the at least two first image acquisition devices;

[0207] The second image acquisition component 10012 includes at least four second image acquisition devices, and the second image acquisition devices are arranged around the body of the excavator; the second image acquisition component 10012 is used to acquire the surrounding original image of the excavator, and the surrounding original image includes at least four second images synchronously acquired by the at least four second image acquisition devices;

[0208] The image processing component 10013 is used to perform smooth stitching processing on the forward-view original image and the surround-view original image in parallel to form a forward-view stitching image stitched by the at least two first images and a surround-view stitching image stitched by the at least four second images;

[0209] The remote operation system 1002 at least includes: an image display component 10021 and an excavator control component 10022;

[0210] The image display component 10021 includes at least one image display device, which is used to display the forward-view stitched image and the surround-view stitched image;

[0211] The excavator control component 10022 is used to control the operation of the excavator according to the control instructions of the remote operator of the excavator; the control instructions are determined by the remote operator of the excavator based on the forward-looking stitched image and the surround-looking stitched image displayed on the image display component 10021.

[0212] The image processing component 10013 provided in this embodiment can also be used to send the forward view stitched image and the surround view stitched image to the remote operation system 1002, so as to display the forward view stitched image and the surround view stitched image on the image display component 10021 of the remote operation system 1002, so that the remote operator of the excavator can issue the control instructions to the excavator control component 10022 according to the forward view stitched image and the surround view stitched image displayed on the image display component 10021.

[0213] The image processing component 10013 provided in this embodiment can also be used to mark the projection position of the bucket in the front-view stitched image.

[0214] The image processing component 10013 provided in this embodiment can also be used to mark a distance scale in the front-view stitched image.

[0215] The image processing component 10013 provided in this embodiment may include: a graphics processor, on which a pre-programmed graphics processing program is constructed to achieve parallel smooth stitching processing of the forward-looking original image and the surround-looking original image.

[0216] The image processing component 10013 provided in this embodiment can also execute the image acquisition method for remote control of the excavator described in the first embodiment of the present application, so that the remote operator can simultaneously see the video of the working scene in front of the excavator composed of the forward-looking stitched image of each frame, and the video of the working scene around the excavator composed of the surround-looking stitched image of each frame on the image display component 10021 of the remote operation system 1002.

[0217] The image display component 10021 can be an image display device, which displays the forward-view stitched image and the surround-view stitched image on the same image display device through a partition display mode or a picture-in-picture display mode. The image display component 10021 can also be two image display devices, which display the forward-view stitched image and the surround-view stitched image on one image display device respectively.

[0218] The forward-looking stitched image displayed on the image display component 10021 may also be a forward-looking stitched image marked with the projection position and distance scale of the bucket, so that the remote operator can obtain more information about the excavator operation scene and control the excavator operation more accurately.

[0219] A third embodiment of the present application provides an excavator, Fig.11 It is a schematic diagram of the structure of the excavator provided in this embodiment.

[0220] like Fig.11As shown, the excavator provided in this embodiment is equipped with a vehicle-mounted system, and the vehicle-mounted system includes: a first image acquisition component 1101, a second image acquisition component 1102, and an image processing component 1103.

[0221] The first image acquisition component 1101 includes at least two first image acquisition devices arranged at the front end of the excavator body, which synchronously acquire at least two first images within the working range of the excavator as forward-looking original images.

[0222] The second image acquisition component 1102 includes at least four second image acquisition devices arranged around the body of the excavator, which acquire at least four second images around the excavator as surround original images.

[0223] The image processing component 1103 includes a graphics processor, on which a pre-programmed graphics processing program is built, which performs parallel smooth stitching processing on the forward-looking original image and the surround-looking original image to form a forward-looking stitched image stitched by the at least two first images and a surround-looking stitched image stitched by the at least four second images.

[0224] The first image acquisition component 1101 provided in this embodiment may include two fisheye cameras arranged at the front end of the excavator body; the two fisheye cameras synchronously acquire two first images as the forward-looking original images; and there is an overlapping area between the two first images acquired by the two fisheye cameras.

[0225] The second image acquisition component 1102 provided in this embodiment may include four fisheye cameras respectively arranged at the front end, rear end, left end and right end of the excavator; the four fisheye cameras synchronously acquire four second images as the surround original images; and there is an overlapping area between the two second images acquired by two adjacent fisheye cameras.

[0226] The image processing component 1103 provided in this embodiment further includes a bucket projection position marking module for marking the projection position of the bucket in the front view stitched image.

[0227] The image processing component 1103 provided in this embodiment further includes a distance scale marking module for marking a distance scale in the front-view stitched image.

[0228] The image processing component 1103 provided in this embodiment further includes an image sending module for sending the forward-view stitched image and the surround-view stitched image to a remote control terminal of the excavator.

[0229] The image processing component 1103 provided in this embodiment can also execute the image acquisition method for remote control of the excavator described in the first embodiment of the present application, so that the remote operator can simultaneously see the video of the working scene in front of the excavator composed of the forward-looking stitched image of each frame, and the video of the working scene around the excavator composed of the surround-looking stitched image of each frame on the screen of the remote control end of the excavator.

[0230] A fourth embodiment of the present application provides an image acquisition device for remote control of an excavator. Fig.12 It is a schematic diagram of the structure of the image acquisition device for remote control of an excavator provided in this embodiment.

[0231] like Fig.12 As shown, the image acquisition device for remote control of an excavator provided in this embodiment includes: a forward-looking original image acquisition unit 1201, a surround-looking original image acquisition unit 1202, and a smoothing splicing processing unit 1203.

[0232] The forward-view original image acquisition unit 1201 is used to acquire a forward-view original image, which includes at least two first images synchronously acquired by at least two first image acquisition devices arranged at the front end of the excavator body.

[0233] The surround view original image acquisition unit 1202 is used to acquire a surround view original image, and the surround view original image includes at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator.

[0234] The smoothing stitching processing unit 1203 is used to perform parallel smoothing stitching processing on the forward-looking original image and the surround-looking original image to obtain a forward-looking stitching image stitched by the at least two first images and a surround-looking stitching image stitched by the at least four second images.

[0235] Optionally, the method further includes: sending the forward view stitched image and the surround view stitched image to a remote control terminal of the excavator, so as to display the forward view stitched image and the surround view stitched image on a screen of the remote control terminal of the excavator, so that a remote operator of the excavator can remotely control the excavator according to the forward view stitched image and the surround view stitched image displayed on the screen.

[0236] Optionally, the method further includes: marking the projection position of the bucket in the forward-view stitched image; sending the forward-view stitched image and the surround-view stitched image marked with the projection position of the bucket to the remote control terminal of the excavator, so as to display the forward-view stitched image and the surround-view stitched image marked with the projection position of the bucket on the screen of the remote control terminal of the excavator.

[0237] Optionally, the method further includes: marking a distance scale in the forward view stitched image; and sending the forward view stitched image and the surround view stitched image marked with the distance scale to a remote control terminal of the excavator, so as to display the forward view stitched image and the surround view stitched image marked with the distance scale on a screen of the remote control terminal of the excavator.

[0238] Optionally, marking the projection position of the bucket in the front-view stitched image includes:

[0239] Obtain a first conversion parameter for converting the bucket coordinate system into the base coordinate system, and a second conversion parameter for converting the bucket coordinate system into the camera coordinate system;

[0240] Determine a third conversion parameter for converting the camera coordinate system into a base coordinate system according to the first conversion parameter and the second conversion parameter;

[0241] Calculate the second coordinate of the projection point of the bucket in the camera coordinate system according to the third conversion parameter and the first coordinate of the bucket in the base coordinate system;

[0242] Determine, according to the second coordinate, a third coordinate of the projection point of the bucket in a pixel coordinate system;

[0243] According to the third coordinate, the projection position of the bucket is marked in the front-view stitched image.

[0244] Optionally, the parallel smooth stitching processing of the forward view original image and the surround view original image includes: using a graphics processor to parallel smooth stitching processing of the forward view original image and the surround view original image, and a pre-programmed graphics processing program is built on the graphics processor.

[0245] Optionally, the smooth stitching process includes: image projection and image fusion.

[0246] Optionally, the performing smoothing stitching processing on the forward-looking original image to obtain a forward-looking stitched image stitched by stitching the at least two first images includes:

[0247] Based on a first projection algorithm, projecting the at least two first images onto a graphic surface to form a front-view combined image;

[0248] Based on a smoothing algorithm, a smoothing fusion process is performed on the overlapping area of ​​the at least two first images projected in the forward-looking combined image to form the forward-looking stitched image.

[0249] Optionally, projecting the at least two first images onto a graphic surface based on a first projection algorithm to form a forward-looking combined image includes:

[0250] respectively calculating the camera parameters corresponding to the at least two first image acquisition devices;

[0251] According to the camera parameters, the at least two first images are projected onto a cylindrical surface to form the forward-looking combined image.

[0252] Optionally, the performing smooth stitching processing on the surround original image to obtain a surround stitched image stitched by stitching the at least four second images includes:

[0253] Based on a second projection algorithm, projecting the at least four second images onto a ground plane to form a surround view combined image;

[0254] Based on the smoothing algorithm, smoothing and fusion processing is performed on the two-by-two overlapping areas of the at least four second images projected in the surround view combined image to form the surround view stitched image.

[0255] Optionally, projecting the at least four second images onto a ground plane based on a second projection algorithm to form a surround view combined image includes:

[0256] respectively calculating the camera parameters corresponding to the at least four second image acquisition devices and the projection parameters corresponding to the at least four second images;

[0257] The at least four second images are projected onto a ground plane according to the camera parameters and the projection parameters to form a surround view combined image.

[0258] Optionally, the smoothing algorithm includes: a multi-band fusion algorithm.

[0259] A fifth embodiment of the present application provides an electronic device. Fig.13 It is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0260] like Fig.13 As shown, the electronic device provided in this embodiment includes: a memory 1301 and a processor 1302.

[0261] The memory 1301 is used to store computer instructions for executing the image acquisition method for remote control of an excavator.

[0262] The processor 1302 is used to execute the computer instructions stored in the memory 1301 to perform the following operations:

[0263] Acquire a forward-view original image, wherein the forward-view original image includes at least two first images synchronously acquired by at least two first image acquisition devices disposed at a front end of a body of the excavator;

[0264] Acquire a surround original image, wherein the surround original image includes at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator;

[0265] The forward-view original image and the surround-view original image are parallelly and smoothly stitched together to obtain a forward-view stitched image stitched together by the at least two first images and a surround-view stitched image stitched together by the at least four second images.

[0266] Optionally, the method further includes: sending the forward view stitched image and the surround view stitched image to a remote control terminal of the excavator, so as to display the forward view stitched image and the surround view stitched image on a screen of the remote control terminal of the excavator, so that a remote operator of the excavator can remotely control the excavator according to the forward view stitched image and the surround view stitched image displayed on the screen.

[0267] Optionally, the method further comprises: marking a projection position of the bucket in the front-view stitched image;

[0268] The forward view stitched image and the surround view stitched image marked with the projection position of the bucket are sent to the remote control terminal of the excavator, so that the forward view stitched image and the surround view stitched image marked with the projection position of the bucket are displayed on the screen of the remote control terminal of the excavator.

[0269] Optionally, the method further comprises: marking a distance scale in the front-view stitched image;

[0270] The forward view stitched image and the surround view stitched image marked with the distance scale are sent to the remote control terminal of the excavator, so that the forward view stitched image and the surround view stitched image marked with the distance scale and the surround view stitched image are displayed on the screen of the remote control terminal of the excavator.

[0271] Optionally, marking the projection position of the bucket in the front-view stitched image includes:

[0272] Obtain a first conversion parameter for converting the bucket coordinate system into the base coordinate system, and a second conversion parameter for converting the bucket coordinate system into the camera coordinate system;

[0273] Determine a third conversion parameter for converting the camera coordinate system into a base coordinate system according to the first conversion parameter and the second conversion parameter;

[0274] Calculate the second coordinate of the projection point of the bucket in the camera coordinate system according to the third conversion parameter and the first coordinate of the bucket in the base coordinate system;

[0275] Determine, according to the second coordinate, a third coordinate of the projection point of the bucket in a pixel coordinate system;

[0276] According to the third coordinate, the projection position of the bucket is marked in the front-view stitched image.

[0277] Optionally, the parallel smooth stitching processing of the forward view original image and the surround view original image includes: using a graphics processor to parallel smooth stitching processing of the forward view original image and the surround view original image, and a pre-programmed graphics processing program is built on the graphics processor.

[0278] Optionally, the smooth stitching process includes: image projection and image fusion.

[0279] Optionally, the performing smoothing stitching processing on the forward-looking original image to obtain a forward-looking stitched image stitched by stitching the at least two first images includes:

[0280] Based on a first projection algorithm, projecting the at least two first images onto a graphic surface to form a front-view combined image;

[0281] Based on a smoothing algorithm, a smoothing fusion process is performed on the overlapping area of ​​the at least two first images projected in the forward-looking combined image to form the forward-looking stitched image.

[0282] Optionally, projecting the at least two first images onto a graphic surface based on a first projection algorithm to form a forward-looking combined image includes:

[0283] respectively calculating the camera parameters corresponding to the at least two first image acquisition devices;

[0284] According to the camera parameters, the at least two first images are projected onto a cylindrical surface to form the forward-looking combined image.

[0285] Optionally, the performing smooth stitching processing on the surround original image to obtain a surround stitched image stitched by stitching the at least four second images includes:

[0286] Based on a second projection algorithm, projecting the at least four second images onto a ground plane to form a surround view combined image;

[0287] Based on the smoothing algorithm, smoothing and fusion processing is performed on the two-by-two overlapping areas of the at least four second images projected in the surround view combined image to form the surround view stitched image.

[0288] Optionally, projecting the at least four second images onto a ground plane based on a second projection algorithm to form a surround view combined image includes:

[0289] respectively calculating the camera parameters corresponding to the at least four second image acquisition devices and the projection parameters corresponding to the at least four second images;

[0290] The at least four second images are projected onto a ground plane according to the camera parameters and the projection parameters to form a surround view combined image.

[0291] Optionally, the smoothing algorithm includes: a multi-band fusion algorithm.

[0292] The sixth embodiment of the present application provides a readable storage medium, which includes readable instructions. When the readable instructions are executed by a processor, they are used to implement the method described in the first embodiment of the present application.

[0293] It should be noted that the relational terms such as "first" and "second" in this article are only used to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "include", "have", "include" and "includes" and other similar forms are the same in meaning, and the end of any one or more items after any of the above words is open-ended, and any of the above nouns does not mean that the one or more items have been exhaustively listed or are limited to the one or more items listed.

[0294] As used herein, unless expressly stated otherwise, the term "or" includes all possible combinations, except those that are not feasible. For example, if it is expressed that a database may include A or B, then unless otherwise specifically stated or not feasible, it may include database A, or B, or A and B. As a second example, if it is expressed that a database may include A, B, or C, then unless otherwise specifically stated or not feasible, the database may include database A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0295] It is worth noting that the above embodiments can be implemented by hardware or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-mentioned computer-readable medium. When the software is executed by a processor, it can execute the above-mentioned disclosed method. The computing unit and other functional units described in the present disclosure can be implemented by hardware or software, or a combination of hardware and software. Those of ordinary skill in the art will also understand that the above-mentioned multiple modules / units can be combined into one module / unit, and each of the above-mentioned modules / units can be further divided into multiple sub-modules / sub-units.

[0296] In the above detailed description, the embodiments have been described with reference to many specific details, which may vary from implementation to implementation. Certain adaptations and modifications may be made to the embodiments. For those skilled in the art, other embodiments may be readily apparent from the specific embodiments disclosed in the present application. This specification and examples are for exemplary purposes only, and the true scope and essence of the present application are described by the claims. The sequence of steps shown in the diagrams is also for the purpose of explanation only and is not intended to be limited to any particular steps or sequence. Therefore, those skilled in the art will appreciate that these steps may be performed in different orders when implementing the same method.

[0297] In the drawings and detailed description of the present application, exemplary embodiments are disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are used, these terms are only general and descriptive and not for the purpose of limitation.

Claims

1. An image acquisition method for remote control of an excavator, characterized in that: The method comprises: Acquire a forward-view original image, wherein the forward-view original image includes at least two first images synchronously acquired by at least two first image acquisition devices disposed at the front end of the vehicle body of the excavator, wherein the first image acquisition device is a fisheye camera; Acquire a surround original image, wherein the surround original image includes at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator; Performing parallel smoothing stitching processing on the forward-looking original image and the surround-looking original image to obtain a forward-looking stitched image stitched by the at least two first images and a surround-looking stitched image stitched by the at least four second images; Marking the projection position of the bucket and the distance scale in the front-view stitched image to indicate the distance between the position where the bucket falls on the ground and the center of the excavator; The step of marking the projection position of the bucket in the front-view stitched image comprises: Acquire a first conversion parameter for converting a bucket coordinate system into a base coordinate system, and a second conversion parameter for converting a bucket coordinate system into a camera coordinate system, wherein the base coordinate system is a boom root coordinate system of the excavator; Determine a third conversion parameter for converting the camera coordinate system into a base coordinate system according to the first conversion parameter and the second conversion parameter; According to the third conversion parameter and the first coordinate of the bucket in the basic coordinate system, the second coordinate of the projection point of the bucket in the camera coordinate system is calculated; specifically: the coordinate of the projection point of the bucket in the basic coordinate system is determined according to the first coordinate of the bucket in the basic coordinate system and the vertical distance from the root of the boom of the excavator to the ground; the second coordinate of the projection point of the bucket in the camera coordinate system is determined according to the coordinate of the projection point of the bucket in the basic coordinate system and the third conversion parameter; Determine, according to the second coordinate, a third coordinate of the projection point of the bucket in a pixel coordinate system, wherein the third coordinate represents a position of the projection point in the forward-looking original image; According to the third coordinate, the projection position of the bucket is marked in the front-view stitched image.

2. The method according to claim 1, characterized in that: The method also includes: sending the forward view stitched image and the surround view stitched image marked with the projection position of the bucket and the distance scale to the remote control terminal of the excavator, so as to display the forward view stitched image and the surround view stitched image marked with the projection position of the bucket and the distance scale on the screen of the remote control terminal of the excavator, so that the remote operator of the excavator can remotely control the excavator according to the forward view stitched image and the surround view stitched image displayed on the screen.

3. The method according to claim 1, characterized in that The parallel smooth splicing processing of the forward-looking original image and the surround-looking original image includes: using a graphics processor to parallel smooth splicing processing of the forward-looking original image and the surround-looking original image, and a pre-programmed graphics processing program is built on the graphics processor.

4. The method according to claim 1, characterized in that: The smooth splicing process includes: image projection and image fusion.

5. The method according to claim 4, characterized in that The step of performing smooth stitching processing on the forward-looking original image to obtain a forward-looking stitched image stitched by stitching the at least two first images comprises: Based on a first projection algorithm, projecting the at least two first images onto a graphic surface to form a front-view combined image; Based on a smoothing algorithm, a smoothing fusion process is performed on the overlapping area of ​​the at least two first images projected in the forward-looking combined image to form the forward-looking stitched image.

6. The method according to claim 5, characterized in that The method of projecting the at least two first images onto a graphic surface based on a first projection algorithm to form a front-view combined image comprises: respectively calculating the camera parameters corresponding to the at least two first image acquisition devices; According to the camera parameters, the at least two first images are projected onto a cylindrical surface to form the forward-looking combined image.

7. The method according to claim 4, characterized in that The step of performing smooth stitching processing on the surround original image to obtain a surround stitched image formed by stitching the at least four second images comprises: Based on a second projection algorithm, projecting the at least four second images onto a ground plane to form a surround view combined image; Based on a smoothing algorithm, smoothing fusion processing is performed on the overlapping areas of the at least four second images projected in the surround view combined image to form the surround view stitched image.

8. The method according to claim 7, characterized in that The step of projecting the at least four second images onto a ground plane based on a second projection algorithm to form a surround view combined image comprises: respectively calculating the camera parameters corresponding to the at least four second image acquisition devices and the projection parameters corresponding to the at least four second images; The at least four second images are projected onto a ground plane according to the camera parameters and the projection parameters to form a surround view combined image.

9. The method according to claim 5 or 7, characterized in that: The smoothing algorithm includes: a multi-band fusion algorithm.

10. A remote control system for an excavator, characterized in that: The system at least includes: an on-board system and a remote operating system; The vehicle-mounted system is configured on an excavator and includes: a first image acquisition component, a second image acquisition component, and an image processing component; The first image acquisition component includes at least two first image acquisition devices, and the first image acquisition devices are arranged at the front end of the body of the excavator; the first image acquisition component is used to acquire a forward-looking original image of the excavator, and the forward-looking original image includes at least two first images synchronously acquired by the at least two first image acquisition devices, and the first image acquisition device is a fisheye camera; The second image acquisition component includes at least four second image acquisition devices, and the second image acquisition devices are arranged around the body of the excavator; the second image acquisition component is used to acquire a surround original image of the excavator, and the surround original image includes at least four second images synchronously acquired by the at least four second image acquisition devices; The image processing component is used to perform smooth stitching processing on the forward-view original image and the surround-view original image in parallel to form a forward-view stitching image formed by stitching the at least two first images and a surround-view stitching image formed by stitching the at least four second images; and is also used to mark the projection position and distance scale of the bucket in the forward-view stitching image to indicate the distance between the position where the bucket falls on the ground and the center of the excavator; The step of marking the projection position of the bucket in the front-view stitched image comprises: Acquire a first conversion parameter for converting a bucket coordinate system into a base coordinate system, and a second conversion parameter for converting a bucket coordinate system into a camera coordinate system, wherein the base coordinate system is a boom root coordinate system of the excavator; Determine a third conversion parameter for converting the camera coordinate system into a base coordinate system according to the first conversion parameter and the second conversion parameter; According to the third conversion parameter and the first coordinate of the bucket in the basic coordinate system, the second coordinate of the projection point of the bucket in the camera coordinate system is calculated; specifically: the coordinate of the projection point of the bucket in the basic coordinate system is determined according to the first coordinate of the bucket in the basic coordinate system and the vertical distance from the root of the boom of the excavator to the ground; the second coordinate of the projection point of the bucket in the camera coordinate system is determined according to the coordinate of the projection point of the bucket in the basic coordinate system and the third conversion parameter; Determine, according to the second coordinate, a third coordinate of the projection point of the bucket in a pixel coordinate system, wherein the third coordinate represents a position of the projection point in the forward-looking original image; According to the third coordinate, the projection position of the bucket is marked in the front view stitching image. The remote operating system at least includes: an image display component and an excavator control component; The image display assembly includes at least one image display device for displaying the forward-view stitched image and the surround-view stitched image marked with the projection position of the bucket and the distance scale; The excavator control component is used to control the excavator operation according to the control instructions of the remote operator of the excavator; the control instructions are determined by the remote operator of the excavator based on the forward-looking stitched image and the surround-looking stitched image marked with the projection position of the bucket and the distance scale displayed on the image display component.

11. The system according to claim 10, characterized in that The image processing component is further used to send the forward-view stitched image and the surround-view stitched image marked with the projection position of the bucket and the distance scale to the remote operating system, so as to display the forward-view stitched image and the surround-view stitched image marked with the projection position of the bucket and the distance scale on the image display component of the remote operating system, so that the remote operator of the excavator can issue the control instructions to the excavator control component according to the forward-view stitched image and the surround-view stitched image displayed on the image display component.

12. The system according to claim 10, characterized in that The image processing component includes: a graphics processor, on which a pre-programmed graphics processing program is constructed to achieve parallel smooth splicing processing of the forward-looking original image and the surround-looking original image.

13. An excavator, characterized in that: A vehicle-mounted system is configured, and the vehicle-mounted system at least includes: a first image acquisition component, a second image acquisition component, and an image processing component; The first image acquisition component includes at least two first image acquisition devices arranged at the front end of the excavator body, synchronously acquiring at least two first images within the working range of the excavator as forward-looking original images, and the first image acquisition device is a fisheye camera; The second image acquisition component includes at least four second image acquisition devices arranged around the body of the excavator, which acquire at least four second images around the excavator as surround original images; The image processing component includes a graphics processor, on which a pre-programmed graphics processing program is built, and parallel smooth stitching processing is performed on the forward-looking original image and the surround-looking original image to form a forward-looking stitched image formed by stitching the at least two first images and a surround-looking stitched image formed by stitching the at least four second images; it also includes a bucket projection position marking module, which marks the projection position of the bucket in the forward-looking stitched image; it also includes a distance scale marking module, which marks a distance scale in the forward-looking stitched image; The step of marking the projection position of the bucket in the front-view stitched image comprises: Acquire a first conversion parameter for converting a bucket coordinate system into a base coordinate system, and a second conversion parameter for converting a bucket coordinate system into a camera coordinate system, wherein the base coordinate system is a boom root coordinate system of the excavator; Determine a third conversion parameter for converting the camera coordinate system into a base coordinate system according to the first conversion parameter and the second conversion parameter; According to the third conversion parameter and the first coordinate of the bucket in the basic coordinate system, the second coordinate of the projection point of the bucket in the camera coordinate system is calculated; specifically: the coordinate of the projection point of the bucket in the basic coordinate system is determined according to the first coordinate of the bucket in the basic coordinate system and the vertical distance from the root of the boom of the excavator to the ground; the second coordinate of the projection point of the bucket in the camera coordinate system is determined according to the coordinate of the projection point of the bucket in the basic coordinate system and the third conversion parameter; Determine, according to the second coordinate, a third coordinate of the projection point of the bucket in a pixel coordinate system, wherein the third coordinate represents a position of the projection point in the forward-looking original image; According to the third coordinate, the projection position of the bucket is marked in the front-view stitched image.

14. The excavator according to claim 13, characterized in that: The first image acquisition component includes two fisheye cameras arranged at the front end of the excavator body; the two fisheye cameras synchronously acquire two first images as the forward-looking original images; and there is an overlapping area between the two first images acquired by the two fisheye cameras.

15. The excavator according to claim 13, characterized in that: The second image acquisition component includes four fisheye cameras respectively arranged at the front end, rear end, left end and right end of the excavator; the four fisheye cameras synchronously acquire four second images as the surround original images; and there is an overlapping area between the two second images acquired by two adjacent fisheye cameras.

16. The excavator according to claim 13, characterized in that: The image processing component also includes an image sending module, which sends the forward-looking stitched image and the surround-looking stitched image marked with the projection position of the bucket and the distance scale to the remote control terminal of the excavator.

17. An image acquisition device for remote control of an excavator, characterized in that: The device comprises: a forward-looking original image acquisition unit, a surround-looking original image acquisition unit, and a smoothing splicing processing unit; The forward-view original image acquisition unit is used to acquire a forward-view original image, wherein the forward-view original image includes at least two first images synchronously acquired by at least two first image acquisition devices arranged at the front end of the vehicle body of the excavator, wherein the first image acquisition device is a fisheye camera; The surround original image acquisition unit is used to acquire a surround original image, wherein the surround original image includes at least four second images synchronously acquired by at least four second image acquisition devices arranged around the body of the excavator; The smoothing splicing processing unit is used to perform parallel smoothing splicing processing on the forward-view original image and the surround-view original image to obtain a forward-view splicing image formed by splicing the at least two first images and a surround-view splicing image formed by splicing the at least four second images; and is also used to mark the projection position and distance scale of the bucket in the forward-view splicing image to indicate the distance between the position where the bucket falls on the ground and the center of the excavator; The step of marking the projection position of the bucket in the front-view stitched image comprises: Acquire a first conversion parameter for converting a bucket coordinate system into a base coordinate system, and a second conversion parameter for converting a bucket coordinate system into a camera coordinate system, wherein the base coordinate system is a boom root coordinate system of the excavator; Determine a third conversion parameter for converting the camera coordinate system into a base coordinate system according to the first conversion parameter and the second conversion parameter; According to the third conversion parameter and the first coordinate of the bucket in the basic coordinate system, the second coordinate of the projection point of the bucket in the camera coordinate system is calculated; specifically: the coordinate of the projection point of the bucket in the basic coordinate system is determined according to the first coordinate of the bucket in the basic coordinate system and the vertical distance from the root of the boom of the excavator to the ground; the second coordinate of the projection point of the bucket in the camera coordinate system is determined according to the coordinate of the projection point of the bucket in the basic coordinate system and the third conversion parameter; Determine, according to the second coordinate, a third coordinate of the projection point of the bucket in a pixel coordinate system, wherein the third coordinate represents a position of the projection point in the forward-looking original image; According to the third coordinate, the projection position of the bucket is marked in the front-view stitched image.

18. An electronic device, characterized in that: include: Memory and processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions to implement the method according to any one of claims 1-9.

19. A readable storage medium having one or more computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method according to any one of claims 1 to 9 is performed.

Images