Panoramic look-around method for working machine and working machine

Abstract

The invention relates to a panoramic look-around method for a working machine and a corresponding working machine. The operation machine comprises an operation mechanism extending outwards from a vehicle body, and a first mechanism camera, a second mechanism camera and at least one other camera are installed on the vehicle body around the vehicle body, wherein the first mechanism camera and the second mechanism camera can shoot at least one part of the operation mechanism. The panoramic looking-around method comprises the following steps: acquiring respective images by using each camera; splicing the images acquired by the first mechanism camera and the second mechanism camera with the images acquired by the other cameras to form a first panoramic image and a second panoramic image; determining a first area and a second area occupied by the working mechanism in the first panoramic image and the second panoramic image respectively; and comparing the first area with the second area and outputting the panoramic image corresponding to the smaller area as the panoramic image of the working machine. According to the panoramic looking-around method, the panoramic image with the minimum shielding or blind area of the working mechanism can be obtained all the time, so that the driving and operation safety of the working machine is improved.

Description

Technical Field

[0001] The present invention relates to the field of visual control of working machinery, and more specifically to a panoramic viewing method and system for working machinery. Background Art

[0002] Panoramic surround view (also known as 360° surround view or 360° panoramic surround view) technology uses multiple cameras installed around the vehicle (usually wide-angle fisheye cameras) to capture images, and uses a special algorithm to stitch and synthesize the images taken by each camera, thereby forming a complete set of 360° panoramic fusion, ultra-wide viewing angle, seamless stitching real-time images around the vehicle and outputting them for display on the on-board screen, allowing the driver to clearly see whether there are pedestrians, moving objects, obstacles, etc. around the vehicle with a wider field of view and understand their relative positions and distances. It is not only very intuitive, but also has almost no blind spots. It can assist the driver to calmly control the vehicle to park or pass through complex roads, effectively reducing the occurrence of scratches, collisions, sinking and other accidents.

[0003] Panoramic surround view technology has not only been widely used in automobiles, but has also been extended to the field of operating machinery or engineering machinery in recent years. It can be used to assist drivers in engineering operations and improve operating safety. The existing panoramic surround view system used in operating machinery (such as excavators and loaders) generally includes four cameras installed in the front, rear, left and right positions of the operating machinery. However, unlike ordinary cars, operating machinery often includes an operating mechanism or working device extending outward from the vehicle body (for example, an excavator includes an outwardly extending boom, dipper and bucket), and the operating mechanism is generally located between the front camera and the right or left camera. In this way, no matter how the installation positions of these cameras are adjusted, the operating mechanism will have different degrees of occlusion when it is in action. This occlusion appears as a blind spot in the generated panoramic surround view image of the operating machinery, which will affect the driver's judgment of the surrounding environment of the operating machinery and may cause danger. Summary of the invention

[0004] The object of the present invention is to solve the above-mentioned problems and / or other drawbacks existing in the existing panoramic view systems for working machines.

[0005] According to one aspect of the present invention, there is provided a panoramic surround view method for a working machine, wherein the working machine comprises a working mechanism extending outward from a body thereof, and a first mechanism camera and a second mechanism camera capable of photographing at least a portion of the working mechanism and at least one remaining camera other than the first and second mechanism cameras are installed on the body around the body, wherein the remaining cameras cannot photograph the working mechanism or can photograph a smaller range of the working mechanism than the first and second mechanism cameras, and the panoramic surround view method comprises the following steps: using the first and second mechanism cameras and the remaining cameras to acquire respective images; generating a first panoramic image using the image acquired by the first mechanism camera and the image acquired by the remaining cameras, and generating a second panoramic image using the image acquired by the second mechanism camera and the image acquired by the remaining cameras; determining a first area occupied by the working mechanism in the first panoramic image and a second area occupied by the working mechanism in the second panoramic image; and comparing the first area with the second area and outputting a panoramic image corresponding to the smaller one of the two as a panoramic image of the working machine.

[0006] According to an exemplary embodiment, the step of determining the first area and the second area includes: selecting a predetermined area including the operating mechanism in the first or second panoramic image, performing edge detection on the operating mechanism part within the predetermined area to determine the outline of the operating mechanism part, and calculating the area enclosed by the outline as the first or second area.

[0007] According to an exemplary embodiment, the first panoramic image and the second panoramic image are two-dimensional bird's-eye view images.

[0008] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. The computer program includes executable instructions. When the executable instructions are executed by a processor, the panoramic view method according to any one of the configurations described above is implemented.

[0009] Another aspect of the present invention provides an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement a panoramic surround view method according to any one of the configurations described above.

[0010] Another aspect of the present invention provides a working machine, comprising: a working mechanism extending outward from its body; a plurality of cameras installed on the body and around the body, the plurality of cameras including a first mechanism camera and a second mechanism camera capable of capturing at least a portion of the working mechanism and at least one remaining camera other than the first and second mechanism cameras, the remaining camera being unable to capture the working mechanism or being able to capture a smaller range of the working mechanism than the first and second mechanism cameras; and the electronic device as described above that is communicatively connected to the plurality of cameras.

[0011] According to an exemplary embodiment, the first mechanism camera and the second mechanism camera are installed on the vehicle body at opposite sides of a vertical plane where the working mechanism is located.

[0012] According to an exemplary embodiment, the vehicle body includes a cab and a cabin, the cab and the front chamber of the cabin are respectively located on opposite sides of a vertical plane where the operating mechanism is located, and the first mechanism camera and the second mechanism camera are respectively installed on the front top of the cab and the front top of the front chamber of the cabin.

[0013] According to an exemplary embodiment, the remaining cameras include a left camera installed on the left top of the cab, and a right camera and a rear camera installed on the right top and rear top of the cabin, respectively.

[0014] According to an exemplary embodiment, the working mechanism includes one or more of a boom, an arm, and a working tool of the working machine.

[0015] Compared with the existing panoramic view systems for working machinery, the panoramic view system according to the present invention adds a camera (referred to as "mechanism camera" in this application) on the body of the working machinery in terms of hardware, that is, it has a total of two mechanism cameras; in terms of software, the added mechanism camera ("second mechanism camera") complements the original mechanism camera ("first mechanism camera"), and the two are used interchangeably, that is, if it is detected that the blind area caused by the occlusion of the working mechanism in the first panoramic image generated by the first mechanism camera and the remaining cameras is smaller than the blind area caused by the occlusion of the working mechanism in the second panoramic image generated by the second mechanism camera and the remaining cameras, then the first panoramic image is used as the panoramic image of the working machinery, for example, displayed on the vehicle-mounted display device of the working machinery; conversely, if it is detected that the blind area of ​​the second mechanism camera is smaller than the blind area of ​​the first mechanism camera, then the second panoramic image is displayed as the panoramic image of the working machinery. The above-mentioned selection and switching between the panoramic images generated by the two mechanism cameras can be repeated in real time during the operation of the operating mechanism, so that a panoramic image with minimal occlusion or blind spot of the operating mechanism can be obtained at each moment and each position of the operating mechanism, thereby reducing the driver's missed judgment or misjudgment of the surrounding environment of the operating machinery due to image blind spots, and improving the safety of driving and operation of the operating machinery. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The features and advantages of the present invention are described in detail below by way of non-limiting examples with reference to the accompanying drawings, wherein the drawings are only schematic and are not necessarily drawn to scale, and furthermore, they only show those parts necessary to illustrate the present invention, while other parts may be omitted or only briefly mentioned. That is, in addition to the parts or elements shown in the drawings, the present invention may also include other parts or elements.

[0017] In the attached picture:

[0018] Figures 1 to 3 They are three-dimensional views of a working machine according to an embodiment of the present invention viewed from three different viewing angles;

[0019] Figure 4 yes Figures 1 to 3 An exemplary panoramic surround image of the work machine is shown;

[0020] Figure 5 is a flow chart of a panoramic viewing method according to an embodiment of the present invention;

[0021] Figure 6 is Figure 5 A schematic illustration of step S130 for determining an operating mechanism shielding area in a panoramic surround view image in the panoramic surround view method shown;

[0022] Figure 7 is a schematic structural diagram of a panoramic view system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Below with reference to the accompanying drawings, exemplary embodiments according to the present invention are described in detail. In the following description, many specific details are set forth so that those skilled in the art can more fully understand the present invention. However, it is obvious to those skilled in the art that the implementation of the present invention may not have some of these specific details. In addition, it should be understood that the present invention is not limited to the specific embodiments introduced. On the contrary, any combination of the features and elements described herein can be considered to implement the present invention, regardless of whether they relate to different embodiments. Therefore, the following aspects, features, embodiments and advantages are only for illustrative purposes and should not be regarded as elements or limitations of the claims, unless clearly stated in the claims.

[0024] Figures 1 to 3 An exemplary work machine 100 to which a panoramic view system and method according to an embodiment of the present invention is applied is shown. The work machine 100 is illustrated as a crawler excavator in this embodiment. The work machine 100 according to the present invention may include a vehicle body and a work mechanism (or working device) 30 extending outward from the vehicle body. The work mechanism 30 may typically include a boom (or large arm) 31, a dipper (or small arm) 32, and a work tool 33 (a bucket for an excavator, or a tool for performing other operations) connected to each other, especially the boom closest to the vehicle body. The vehicle body may include a main frame and a cab 10 and a cabin 20 supported on the main frame. The main frame of the excavator may be constructed, for example, as a rotatable platform, capable of carrying the work mechanism 30, the cab 10, and the cabin 20 for overall rotation so as to facilitate operations in a sufficiently wide range of directions. The cabin 20 may be composed of a plurality of compartments or boxes, such as an engine compartment for receiving the engine system of the working machine, a hydraulic compartment for accommodating the hydraulic system for actuating the working mechanism 30, and a fuel tank for filling fuel. A walking device 40 (in this embodiment, a crawler assembly) for supporting and moving the working machine 100 is also arranged under the vehicle body.

[0025] Terms indicating directions or positional relationships such as "front", "rear", "left", "right", "up", "down", "top", "bottom", "vertical", "horizontal" and the like mentioned in this application are relative to the body of the working machine 100, or from the perspective of a driver normally sitting in the cab. Figures 1 to 3In the coordinate system shown in FIG. 1 , F indicates the forward direction, L indicates the left direction, and U indicates the upward direction. The working mechanism 30 generally extends forward from the vehicle body, is located in a vertical plane (FU plane), can swing up and down in the vertical plane, and can also rotate with the rotation of the vehicle body in a horizontal plane (FL plane).

[0026] In order to generate a panoramic surround image of the working machine 100, the working machine 100 may include a plurality of cameras mounted on the body of the working machine 100. In the present application, the term "camera" should be understood as referring to any image acquisition device that can capture static or dynamic images / videos within a certain viewing angle range, and generally, a wide-angle fisheye camera may be used, for example. The images captured by each of the plurality of cameras can be synthesized into a 360° panoramic surround image (hereinafter referred to as a "panoramic image") around the working machine 100 through stitching processing. The technology of stitching images from multiple cameras into a panoramic image is known in the art, and generally includes camera calibration, de-distortion correction, panoramic model stitching, etc., which will not be described in detail here. As an exemplary configuration of the plurality of cameras, the working machine 100 may include: Figures 1 to 3 The front camera 51, the left camera 52, the right camera 53 and the rear camera 54 are schematically indicated by blank triangles respectively, wherein the front camera 51 is, for example, installed on the front top of the cab 10 (such as the position close to the midpoint of the top edge on the front side of the cab) to mainly shoot the front area of ​​the vehicle body, the left camera 52 is, for example, installed on the left top of the cab 10 (such as the position close to the midpoint of the top edge on the left side surface of the cab) to mainly shoot the left area of ​​the vehicle body, the right camera 53 is, for example, installed on the right top of the cabin 20 (such as the position close to the midpoint of the top edge on the right side surface of the cabin) to mainly shoot the right area of ​​the vehicle body, and the rear camera 54 is, for example, installed on the rear top of the cabin 20 (such as the position close to the midpoint of the top edge on the rear side of the cabin) to mainly shoot the rear area of ​​the vehicle body. Among the above-mentioned four cameras, since the operating mechanism 30 is located in the front area of ​​the vehicle body, the operating mechanism range captured by the front camera 51 is the largest, and the other three cameras either cannot capture the operating mechanism 30 (for the rear camera or the left or right camera with a normal or small shooting angle), or the operating mechanism range that can be captured is small (for the left or right camera with a very large shooting angle).

[0027] The configuration of the above four cameras is also a typical configuration of the panoramic view system of the existing working machinery. An exemplary panoramic image generated by the four cameras can be shown as follows: Figure 4 As shown (it should be noted that Figure 4 is a grayscale image, but the panoramic image can actually be a color image that reflects the real world). Figure 4The left part of the image is a two-dimensional bird's-eye view image (plane top view). Figure 4 The right part of the image is a three-dimensional stereoscopic image. Since the working mechanism 30 of the working machine 100 is located between the front camera 51 and the right camera 53, no matter how the installation positions of these cameras are adjusted, the working mechanism 30 will be blocked to varying degrees when in motion. This blockage appears as a blind spot in the panoramic view image of the working machine (see Figure 4 The area on the left and right sides of the operating mechanism (the area framed by the rectangular solid line and including the blocked part of the operating mechanism) may affect the driver's judgment of the surrounding environment of the operating machinery and thus cause driving or operating hazards.

[0028] In order to reduce the risk of the above-mentioned occlusion or blind spot generated by the working mechanism to the working machine, the panoramic view system adopted by the working machine of the present invention adds a switching camera 55 on the basis of the above-mentioned four cameras, which is complementary to the front camera 51 and switchable (the method of use is described in detail below). Like the front camera 51, it is also used to mainly shoot the front area of ​​the vehicle body, so that it can also shoot a larger range of the working mechanism compared with the other three cameras 52, 53, and 54. For this reason, the switching camera 55 can also be installed on the front side of the vehicle body. For example, in the illustrated embodiment, the cabin or box room part of the cabin 20 close to the front side of the vehicle body (referred to as the "front room" of the cabin in this application) and the cab 10 are respectively located on opposite sides of the vertical plane (i.e., the FU plane) where the working mechanism 30 is located, and the switching camera 55 can be installed on the front top of the front room of the cabin 20. More specifically, the front room of the cabin may, for example, include a diesel tank 21 and a tool box 22, and the switching camera 55 can be as shown. Figures 1 to 3 As shown, it is installed on the front side of the diesel tank 21 near the midpoint of its top edge. Of course, the installation position of the switching camera 55 is not limited to this, but can also be appropriately adjusted according to the specific configuration of the vehicle body of different working machinery, as long as it can capture at least a part of the working mechanism 30 and can capture a larger range of the working mechanism than the remaining cameras except the front camera 51. In addition, the installation position of the front camera 51 also follows this principle. Therefore, the front camera 51 and the switching camera 55 have the same working properties, and can also be generally referred to as the first mechanism camera and the second mechanism camera in this application, respectively. The "first" and "second" here are only distinguished in name for the convenience of description, and do not represent any relative order or importance difference between the two mechanism cameras.

[0029] Next, refer to Figure 5 The following describes a panoramic view method according to an embodiment of the present invention, that is, a method for generating a panoramic image of a working machine using a panoramic view system according to the present invention. Figure 5As shown, in a possible implementation, the panoramic view method according to the present invention may include: step S110 of using each camera to shoot its own image; step S120 of generating first and second panoramic images corresponding to the first and second mechanism cameras respectively; step S130 of respectively determining the first and second areas occupied by the working mechanism in the first and second panoramic images; and step S140 of comparing the first area with the second area and using the panoramic image corresponding to the smaller area as the panoramic image of the working machine.

[0030] In step S110, a plurality of cameras (such as the cameras 51, 52, 53, 54, and 55 described above) installed around the body of the working machine 100 are used to obtain respective images. More specifically, each camera can be made to take pictures in real time to obtain respective instantaneous static images, or each camera can also record respective dynamic images in real time and capture instantaneous images from the dynamic images.

[0031] In step S120, a first panoramic image is generated using the image acquired by the first mechanism camera (front camera 51) and the images acquired by the remaining cameras (left camera 52, right camera 53 and rear camera 54) except the first mechanism camera and the second mechanism camera (switching camera 55), and a second panoramic image is generated using the image acquired by the second mechanism camera and the images acquired by the remaining cameras. The generation of the panoramic image can be performed using image stitching processing known in the art as described above, generally including camera calibration, de-distortion correction, panoramic model stitching, etc. The generation of the first panoramic image and the generation of the second panoramic image can be performed simultaneously or in any order. The generated first and second panoramic images are similar to Figure 4 In the panoramic image shown, there may be an obstruction or blind area caused by the operating mechanism 30.

[0032] In step S130, a first area occupied by the working mechanism 30 in the first panoramic image generated in step S120 and a second area occupied by the working mechanism 30 in the second panoramic image generated in step S120 are determined. This step can be implemented, for example, by computer image processing and recognition techniques, typically such as edge detection. Figure 6 The edge detection is described in more detail. When implementing step S130 using edge detection technology, a predetermined area including the operating mechanism 30 may be selected in the first or second panoramic image. Figure 4The two-dimensional bird's-eye view image of the left part in the image is taken as an example of the first or second panoramic image, and the area framed by a rectangular solid line and including the occluded part of the operating mechanism can be taken as the above-mentioned predetermined area, and the predetermined area can be selected by defining a range with an estimated position and size in the first or second panoramic image, wherein the range of the estimated position and size can ensure that the operating mechanism in the image is covered. Figure 6 The (a) portion of FIG. 1 shows the selected predetermined area, which includes the operating mechanism occlusion portion. Next, edge detection can be performed on the operating mechanism portion within the predetermined area (before edge detection, the edge detection can be performed on the operating mechanism portion). Figure 6 The color image shown in part (a) is actually converted into a grayscale image) to determine the outline of the operating mechanism part, such as Figure 6 As shown in part (b) of FIG. 1 , edge detection can be implemented using various edge detection algorithms known in the art. Finally, the area enclosed by the determined contour is calculated as the first or second area, as shown in FIG. Figure 6 As shown in part (c) of .

[0033] In step S140, the first area determined in step S130 is compared with the second area, and the panoramic image corresponding to the smaller one of the two is output as the panoramic image of the working machine 100. In other words, if the first area is smaller than the second area, it means that the occlusion or blind area generated by the working mechanism 30 in the first panoramic image generated by the image of the first mechanism camera (front camera 51) and the images of the remaining cameras is smaller than the occlusion or blind area generated by the working mechanism 30 in the second panoramic image generated by the image of the second mechanism camera (switching camera 55) and the images of the remaining cameras, then the panoramic image generated by the image of the first mechanism camera is selected; conversely, if the first area is larger than the second area, the panoramic image generated by the image of the second mechanism camera is selected. In this way, it can be ensured that the occlusion or blind area generated by the working mechanism 30 in the panoramic image of the working machine 100 is the smallest. The specific output method of the panoramic image of the working machine can be, for example, displaying the panoramic image on the vehicle-mounted display device of the working machine 100, and of course it can also be called to other occasions where the panoramic image of the working machine needs to be obtained.

[0034] The above steps S110 to S140 can be continuously repeated in real time during the movement process of the working mechanism 30 (such as the lifting or lowering process). Accordingly, the imaging mode of the panoramic image may be switched back and forth between the first and second mechanism cameras, so that a panoramic image with minimal occlusion or blind spots of the working mechanism can be obtained at each moment and each position of the working mechanism, thereby reducing the driver's missed judgment or misjudgment of the surrounding environment of the working machine due to the image blind spots, and improving the safety of driving and operation of the working machine.

[0035] Figure 7 A schematic structural diagram of an exemplary configuration of a panoramic surround view system 200 according to the present invention is shown. The system includes the above-mentioned multiple cameras 51, 52, 53, 54, 55 installed around the body of the working machine 100, and an electronic device 400. The electronic device 400 may include a processor 410, and a memory 420 for storing executable instructions of the processor 410. Among them, the processor 410 is configured to execute the various steps of the panoramic surround view method according to any of the above-mentioned embodiments by executing the executable instructions. The electronic device 400 can be expressed in the form of a general-purpose computing device (such as a high-performance computer or controller on the vehicle of the working machine 100). The electronic device 400 is connected in communication with the above-mentioned multiple cameras so as to control these cameras to obtain corresponding images and receive the acquired images from these cameras. The panoramic surround view system 200 may also include a vehicle-mounted display device 15 connected in communication with the electronic device 400, and the electronic device 400 can output the panoramic image obtained by the above-mentioned panoramic surround view method to the vehicle-mounted display device 15 for display. Optionally, the electronic device 400 may also communicate with one or more other devices (such as various vehicle-mounted sensors, Internet of Vehicles devices, cloud servers, etc.) inside or outside the working machine through various communication ports 300 to obtain any required information.

[0036] The present invention also provides a working machine, which includes a working mechanism extending outward from its body, the above-mentioned multiple cameras installed on the body around the body, and the above-mentioned electronic device 400 communicatively connected to these cameras.

[0037] The present invention also provides a computer-readable storage medium on which a computer program is stored, the program including executable instructions, which can implement the steps of the panoramic surround view method according to any of the above embodiments when the executable instructions are executed by, for example, a processor. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps of various exemplary embodiments of the panoramic surround view method according to the present invention.

[0038] The program product for implementing the above-mentioned panoramic surround view method according to an embodiment of the present invention may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device such as a vehicle-mounted computer. However, the program product of the present invention is not limited thereto, and in this context, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0039] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0040] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0041] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0042] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the panoramic surround view method according to the embodiment of the present invention.

[0043] Industrial Applicability

[0044] When the operating machine 100 is working, the operating mechanism 30 performs a lifting or lowering action, for example. During this process, the above-mentioned electronic device 400 continuously and repeatedly executes the panoramic view method according to any of the above-mentioned embodiments of the present invention in real time, thereby controlling the first and second mechanism cameras and the remaining cameras to obtain their respective images and generate corresponding first and second panoramic images, and according to the size of the occlusion or blind spot generated by the operating mechanism in the first and second panoramic images, the imaging mode of the panoramic image is selectively switched back and forth between the first and second mechanism cameras, so that at each moment and at each position of the operating mechanism, a panoramic image with minimal occlusion or blind spot of the operating mechanism can be obtained, thereby reducing the driver's missed judgment or misjudgment of the surrounding environment of the operating machine due to the image blind spot, and improving the safety of driving and operation of the operating machine.

[0045] The working machine 100 is not limited to the crawler excavator shown in the illustrated embodiment, but may also be any other type of working machine (e.g., wheel loader, etc.) that performs operations related to a specific industry (e.g., mining, construction, agriculture, transportation, etc.) and works and travels in various working environments (e.g., mines, construction sites, farms, roads, etc.), as long as the working machine has a working mechanism that extends outward from its body and may cause occlusion or blind spots in the panoramic image. There is no particular restriction on the specific number of the above-mentioned multiple cameras installed on the working machine, as long as they can collaborate to generate a panoramic image of the working machine. Among the multiple cameras, there are at least two cameras that can capture at least a portion of the working mechanism, and the two cameras that can capture the largest range of the working mechanism are selected as the first and second mechanism cameras.

[0046] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing what is disclosed herein. This disclosure is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed herein. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the appended claims.

Claims

1. A panoramic surround view method for a working machine (100), the working machine comprising a working mechanism (30) extending outward from a vehicle body thereof, a first mechanism camera (51) and a second mechanism camera (55) capable of photographing at least a portion of the working mechanism (30) and at least one other camera (52, 53, 54) other than the first and second mechanism cameras being mounted on the vehicle body and surrounding the vehicle body, the other cameras being unable to photograph the working mechanism (30) or being able to photograph a smaller range of the working mechanism than the first and second mechanism cameras, the panoramic surround view method comprising the following steps: Utilizing the first and second mechanical cameras (51, 55) and the remaining cameras (52, 53, 54) to acquire respective images; Generate a first panoramic image using the image acquired by the first mechanism camera (51) and the images acquired by the remaining cameras (52, 53, 54), and generate a second panoramic image using the image acquired by the second mechanism camera (55) and the images acquired by the remaining cameras (52, 53, 54); determining a first area occupied by the operating mechanism (30) in the first panoramic image and a second area occupied by the operating mechanism (30) in the second panoramic image; and The first area and the second area are compared and a panoramic image corresponding to the smaller one of the two areas is output as a panoramic image of the working machine (100).

2. The panoramic viewing method according to claim 1, characterized in that: The steps of determining the first area and the second area include: selecting a predetermined area including the operating mechanism (30) in the first or second panoramic image, performing edge detection on the operating mechanism part within the predetermined area to determine the outline of the operating mechanism part, and calculating the area enclosed by the outline as the first or second area.

3. The panoramic viewing method according to claim 1 or 2, characterized in that: The first panoramic image and the second panoramic image are two-dimensional bird's-eye view images.

4. A computer-readable storage medium having a computer program stored thereon, the computer program comprising executable instructions, and when the executable instructions are executed by a processor, the panoramic surround view method according to any one of claims 1 to 3 is implemented.

5. An electronic device (400), comprising: Processor (410); as well as a memory (420) for storing executable instructions of the processor; The processor (410) is configured to execute the executable instructions to implement the panoramic view method according to any one of claims 1 to 3.

6. A working machine (100), comprising: An operating mechanism (30) extending outward from the vehicle body; A plurality of cameras are installed on the vehicle body around the vehicle body, the plurality of cameras comprising a first mechanism camera (51) and a second mechanism camera (55) capable of photographing at least a portion of the operating mechanism, and at least one remaining camera (52, 53, 54) other than the first and second mechanism cameras, the remaining camera being unable to photograph the operating mechanism (30) or being able to photograph a smaller range of the operating mechanism than the first and second mechanism cameras; as well as An electronic device (400) according to claim 5, communicatively connected to the plurality of cameras.

7. The working machine (100) according to claim 6, characterized in that: The first mechanism camera (51) and the second mechanism camera (55) are installed on the vehicle body on opposite sides of a vertical plane where the operating mechanism (30) is located.

8. The working machine (100) according to claim 7, characterized in that: The vehicle body comprises a cab (10) and a cabin (20), wherein the cab and the front chambers (21, 22) of the cabin are respectively located on opposite sides of a vertical plane where the operating mechanism (30) is located, and the first mechanism camera (51) and the second mechanism camera (55) are respectively installed on the front top of the cab (10) and the front top of the front chamber (21, 22) of the cabin (20).

9. The working machine (100) according to claim 8, characterized in that: The remaining cameras include a left camera (52) installed on the left top of the cab (10), and a right camera (53) and a rear camera (54) installed on the right top and rear top of the cabin (20), respectively.

10. The working machine (100) according to any one of claims 6 to 9, characterized in that: The working mechanism (30) includes one or more of a boom (31), an arm (32) and a working tool (33) of the working machine.

Images