Display system and display method

Abstract

To provide an operator of a work machine with information on the situation around the work machine even when an event occurs in which an image captured by a visible light imaging device becomes unclear.SOLUTION: A display system includes a visible light image acquisition unit for acquiring a visible light image of a first object captured by a visible light imaging device, an infrared image acquisition unit for acquiring an infrared image of a second object captured by an infrared imaging device, a visible light distance calculation unit for calculating a visible light distance indicating the distance from the visible light imaging device to the first object for each of a plurality of first sectional areas, an infrared distance calculation unit for calculating an infrared distance indicating the distance from the infrared imaging device to the second object for each of a plurality of second sectional areas, a determination unit for determining whether the difference between the visible light distance and the infrared distance for each of the corresponding first and second divided areas is equal to or greater than a distance threshold value, a combination unit for combining the second sectional area where the difference is equal to or greater than the distance threshold value with the visible light image to generate a composite image, and a display output unit for outputting the composite image such that the composite image is displayed on a display device.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a display system and a display method.

Background Art

[0002] In the technical field related to display systems, an image processing apparatus as disclosed in Patent Document 1 is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technical field related to working machines, a technique of imaging the periphery of a working machine with a visible light imaging device is known. By providing an image captured by the visible light imaging device to an operator of the working machine, the operator can confirm the situation around the working machine. There is a possibility that dust may be generated due to the operation of the working machine. Also, there is a possibility that the operation of the working machine may be carried out in a situation where fog is present. When dust or fog occurs, the image captured by the visible light imaging device may become unclear. Also, even when the operation of the working machine is carried out at night, the image captured by the visible light imaging device may become unclear. Also, even when imaging an imaging target in a backlight state, the image captured by the visible light imaging device may become unclear.

[0005] An object of the present disclosure is to provide the situation around a working machine to an operator of the working machine even when an event occurs in which an image captured by a visible light imaging device becomes unclear.

Means for Solving the Problems

[0006] According to this disclosure, a display system is provided, comprising: a visible light image acquisition unit that acquires a visible light image showing an image of a first target captured by a visible light imaging device provided on a work machine; an infrared image acquisition unit that acquires an infrared image showing an image of a second target captured by an infrared imaging device provided on the work machine; a visible light distance calculation unit that calculates a visible light distance indicating the distance from the visible light imaging device to the first target for each of a plurality of first partition regions defined in the visible light image; an infrared distance calculation unit that calculates an infrared distance indicating the distance from the infrared imaging device to the second target for each of a plurality of second partition regions defined in the infrared image to correspond to the first partition regions; a determination unit that determines whether the difference between the visible light distance and the infrared distance for each corresponding first and second partition regions is greater than or equal to a distance threshold; a synthesis unit that synthesizes the second partition region whose difference is greater than or equal to the distance threshold with the visible light image to generate a composite image; and a display output unit that outputs the composite image so that the composite image can be displayed on a display device. [Effects of the Invention]

[0007] According to this disclosure, even if an event occurs in which the image captured by the visible light imaging device becomes unclear, the operator of the work machine can be provided with information about the surrounding conditions of the work machine. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing a remote control system for a work machine according to an embodiment. [Figure 2] Figure 2 is a perspective view showing a work machine according to an embodiment. [Figure 3] Figure 3 is a perspective view showing a visible light imaging device and a far-infrared imaging device according to an embodiment. [Figure 4] Figure 4 is a functional block diagram showing a remote control system for a work machine according to an embodiment. [Figure 5] Figure 5 is a schematic diagram illustrating the visible light imaging device and the infrared imaging device according to the embodiment. [Figure 6]Figure 6 is a schematic diagram illustrating the visible light image and infrared image according to the embodiment. [Figure 7] Figure 7 is a schematic diagram illustrating the visible light imaging device and the infrared imaging device according to the embodiment. [Figure 8] Figure 8 is a schematic diagram illustrating the visible light image and infrared image according to the embodiment. [Figure 9] Figure 9 is a schematic diagram illustrating the method for generating a composite image according to the embodiment. [Figure 10] Figure 10 is a schematic diagram illustrating the method for generating a composite image according to the embodiment. [Figure 11] Figure 11 is a schematic diagram showing a composite image and a guideline image displayed in the display device according to the embodiment. [Figure 12] Figure 12 is a flowchart showing the display method according to the embodiment. [Figure 13] Figure 13 is a block diagram showing a computer system according to an embodiment. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings, but this disclosure is not limited to these embodiments. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0010] [Remote control system] Figure 1 is a schematic diagram showing a remote control system 100 for a work machine 1 according to an embodiment. The remote control system 100 remotely controls the work machine 1 located at the work site. At least a part of the remote control system 100 is located in a remote control room 200 at the remote control site. The remote control system 100 comprises a remote control device 40, a display device 50, and a control device 60.

[0011] The remote control device 40 is arranged in the remote control room 200 outside the working machine 1. The remote control device 40 is operated by an operator in the remote control room 200. The operator can operate the remote control device 40 while sitting on the operator's seat 45.

[0012] The display device 50 is arranged in the remote control room 200 outside the working machine 1. The display device 50 displays an image of the work site. The image of the work site includes an image of a predetermined range around the working machine 1. The image of the predetermined range around the working machine 1 includes at least an image of the work target of the working machine 1. The work target of the working machine 1 includes the construction target of the working machine 1.

[0013] The display device 50 includes a flat panel display such as a liquid crystal display (LCD) or an organic EL display (OELD). In an embodiment, the display device 50 includes a plurality of flat panel displays arranged adjacent to each other. The display device 50 may be composed of one flat panel display.

[0014] The operator operates the remote control device 40 while checking the image of the work site displayed on the display device 50. The working machine 1 is remotely controlled by the remote control device 40.

[0015] The control device 60 is arranged in the remote control room 200 outside the working machine 1. The control device 60 includes a computer system.

[0016] The working machine 1 includes a control device 300. The control device 300 includes a computer system.

[0017] The control device 60 and the control device 300 communicate with each other via a communication system 400. Examples of the communication system 400 include the Internet, a local area network (LAN), a mobile phone communication network, and a satellite communication network.

[0018] [Working machinery] Figure 2 is a perspective view showing a work machine 1 according to an embodiment. In this embodiment, the work machine 1 is a hydraulic excavator. The work machine 1 operates at the work site.

[0019] As shown in Figure 2, the work machine 1 comprises a traveling body 2, a rotating body 3 supported by the traveling body 2, a work machine 4 supported by the rotating body 3, a hydraulic cylinder 5 that drives the work machine 4, a visible light imaging device 20, and an infrared imaging device 30.

[0020] The mobile unit 2 is capable of traveling while supporting the slewing unit 3. The slewing unit 3 is capable of rotatable around the slewing axis RX while being supported by the mobile unit 2. The work equipment 4 includes a boom 4A connected to the slewing unit 3, an arm 4B connected to the boom 4A, a bucket 4C connected to the arm 4B, and a hydraulic cylinder 5, which includes a boom cylinder 5A for driving the boom 4A, an arm cylinder 5B for driving the arm 4B, and a bucket cylinder 5C for driving the bucket 4C.

[0021] With respect to the slewing axis RX, the direction in which the work implement 4 is located is forward, and the opposite direction of forward is backward. With respect to the slewing axis RX, one of the left and right directions is to the right, and the opposite direction of right is to the left. The direction away from the ground contact surface of the traveling body 2 is upward, and the opposite direction of upward is downward.

[0022] The visible light imaging device 20 images the work site. The visible light imaging device 20 is installed on the work machine 1. In one embodiment, the visible light imaging device 20 is installed on the rotating body 3. The visible light imaging device 20 images a predetermined area around the work machine 1. The visible light imaging device 20 acquires an image in the visible light wavelength range. The visible light wavelength range is, for example, 360 nm to 830 nm.

[0023] The infrared imaging device 30 images the work site. The infrared imaging device 30 is installed on the work machine 1. In one embodiment, the infrared imaging device 30 is installed on the rotating body 3. The infrared imaging device 30 images a predetermined range around the work machine 1. The infrared imaging device 30 acquires an image of the infrared spectral range. The infrared spectral range is 780 nm to 100 μm. In one embodiment, the infrared imaging device 30 acquires an image of the far-infrared spectral range. The spectral range of the infrared imaging device 30 is, for example, 7.5 μm to 14 μm.

[0024] The visible light imaging device 20 and the infrared imaging device 30 each capture images of objects present around the work machine 1. The objects to be captured are objects. Examples of objects to be captured by the visible light imaging device 20 and the infrared imaging device 30 include the work area of ​​work machine 1, the excavation area of ​​work machine 4, structures present at the work site, at least a part of work machine 1, work machines other than work machine 1, and people (workers) working at the work site.

[0025] Each of the visible light imaging device 20 and the infrared imaging device 30 includes an optical system and an image sensor that receives light that has passed through the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0026] The imaging range of the visible light imaging device 20 and at least a portion of the imaging range of the infrared imaging device 30 coincide. The imaging range of the visible light imaging device 20 includes the field of view of the optical system of the visible light imaging device 20. The imaging range of the infrared imaging device 30 includes the field of view of the optical system of the infrared imaging device 30. In this embodiment, the imaging range of the visible light imaging device 20 and the imaging range of the infrared imaging device 30 coincide. However, the imaging range of the visible light imaging device 20 and a portion of the imaging range of the infrared imaging device 30 may coincide.

[0027] [Visible light imaging device and far-infrared imaging device] Figure 3 is a perspective view showing the visible light imaging device 20 and the infrared imaging device 30 according to the embodiment. As shown in Figure 3, the visible light imaging device 20 and the infrared imaging device 30 are each positioned on the upper part of the front of the rotating body 3. The visible light imaging device 20 and the infrared imaging device 30 each image the area in front of the rotating body 3. In this embodiment, the predetermined area around the work machine 1 imaged by the visible light imaging device 20 and the infrared imaging device 30 is the area in front of the rotating body 3.

[0028] In one embodiment, the visible light imaging device 20 includes a first visible light camera 21 and a second visible light camera 22. Each of the first visible light camera 21 and the second visible light camera 22 acquires an image in the wavelength range of visible light. The first visible light camera 21 and the second visible light camera 22 are arranged in the left-right direction.

[0029] In one embodiment, the infrared imaging device 30 includes a first infrared camera 31 and a second infrared camera 32. Each of the first infrared camera 31 and the second infrared camera 32 acquires an image of the infrared spectral range. In one embodiment, each of the first infrared camera 31 and the second infrared camera 32 is a far-infrared camera. The first infrared camera 31 and the second infrared camera 32 are arranged in the left-right direction.

[0030] The first infrared camera 31 is positioned to the left of the first visible light camera 21. The second infrared camera 32 is positioned to the right of the second visible light camera 22. The first visible light camera 21 and the second visible light camera 22 are positioned between the first infrared camera 31 and the second infrared camera 32.

[0031] The first infrared camera 31 and the second infrared camera 32 may be positioned between the first visible light camera 21 and the second visible light camera 22.

[0032] The imaging range of the first visible light camera 21, the imaging range of the second visible light camera 22, the imaging range of the first infrared camera 31, and the imaging range of the second infrared camera 32 are the same. The first visible light camera 21, the second visible light camera 22, the first infrared camera 31, and the second infrared camera 32 simultaneously image the area in front of the rotating body 3.

[0033] Furthermore, it is sufficient if the imaging range of the first visible light camera 21 and at least a portion of the imaging range of the second visible light camera 22 coincide. It is sufficient if the imaging range of the first infrared camera 31 and at least a portion of the imaging range of the second infrared camera 32 coincide. It is sufficient if the imaging range of the visible light imaging device 20 and a portion of the imaging range of the infrared imaging device 30 coincide.

[0034] In the following description, images captured by the visible light imaging device 20 will be referred to as visible light images Ga, and images captured by the infrared imaging device 30 will be referred to as infrared images Gb.

[0035] [Display System] Figure 4 is a functional block diagram showing the remote control system 100 of the work machine 1 according to the embodiment. As shown in Figure 4, the remote control system 100 has a display system 10 that displays images of the work site. The remote control system 100 also includes a communication device 6 located at the remote control site, a control device 60 connected to the communication device 6, a remote control device 40 connected to the control device 60, and a display device 50 connected to the control device 60. The remote control system 100 also includes a communication device 7 located on the work machine 1, a control device 300 connected to the communication device 7, a visible light imaging device 20 connected to the control device 300, an infrared imaging device 30 connected to the control device 300, a traveling body 2 controlled by the control device 300, a rotating body 3 controlled by the control device 300, and a hydraulic cylinder 5 controlled by the control device 300. The display system 10 includes the visible light imaging device 20, the infrared imaging device 30, the control device 60, and the display device 50.

[0036] The control device 300 includes a traveling body control unit 301, a rotating body control unit 302, a work machine control unit 303, and an image output unit 304.

[0037] The vehicle control unit 301 receives an operation signal from the remote control device 40 transmitted from the control device 60. Based on the operation signal from the remote control device 40, the vehicle control unit 301 outputs a control signal to control the operation of the vehicle 2.

[0038] The slewing body control unit 302 receives operation signals from the remote control device 40 transmitted from the control device 60. Based on the operation signals from the remote control device 40, the slewing body control unit 302 outputs a control signal to control the movement of the slewing body 3.

[0039] The work equipment control unit 303 receives operation signals from the remote control device 40 transmitted from the control device 60. Based on the operation signals from the remote control device 40, the work equipment control unit 303 outputs control signals to control the operation of the work equipment 4. The control signals for controlling the work equipment 4 include control signals for controlling the hydraulic cylinder 5.

[0040] The image output unit 304 outputs visible light image data showing the visible light image Ga captured by the visible light imaging device 20. The image output unit 304 also outputs infrared image data showing the infrared image Gb captured by the infrared imaging device 30.

[0041] Communication device 7 communicates with communication device 6 via communication system 400. Communication device 7 receives operation signals from the remote control device 40 transmitted from the control device 60 via communication device 6 and outputs them to the control device 300. Communication device 7 transmits visible light image data and infrared image data output from the image output unit 304 to communication device 6. Communication device 7 includes encoders for compressing the visible light image data and infrared image data, respectively. The visible light image data and infrared image data are transmitted from communication device 7 to communication device 6 in a compressed state.

[0042] Communication device 6 communicates with communication device 7 via communication system 400. Communication device 6 transmits operation signals generated by the operation of remote control device 40 to communication device 7. Communication device 6 receives visible light image data and infrared image data transmitted from control device 300 via communication device 7 and outputs them to control device 60. Communication device 6 includes decoders that restore the compressed visible light image data and infrared image data, respectively. The restored visible light image data and infrared image data are output from communication device 6 to control device 60.

[0043] The control device 60 includes an operation signal output unit 61, a visible light image acquisition unit 62, an infrared image acquisition unit 63, a visible light distance calculation unit 64, an infrared distance calculation unit 65, a determination unit 66, a synthesis unit 67, a guideline generation unit 68, and a display output unit 69.

[0044] The operation signal output unit 61 outputs an operation signal for remotely controlling the work machine 1. When the remote control device 40 is operated by the operator, an operation signal for remotely controlling the work machine 1 is generated. The operation signal output unit 61 outputs the operation signal of the remote control device 40. The communication device 6 transmits the operation signal output from the operation signal output unit 61 to the communication device 7.

[0045] The visible light image acquisition unit 62 acquires a visible light image Ga showing an image of a first target captured by the visible light imaging device 20. The visible light image acquisition unit 62 acquires the visible light image Ga by acquiring the visible light image data restored by the communication device 6. The first target includes the object to be captured that is within the imaging range of the visible light imaging device 20.

[0046] The infrared image acquisition unit 63 acquires an infrared image Gb showing the image of the second target captured by the infrared imaging device 30. The infrared imaging device 30 acquires the infrared image Gb by acquiring the infrared image data restored by the communication device 6. The second target includes the target to be captured that is within the imaging range of the infrared imaging device 30.

[0047] The visible light distance calculation unit 64 calculates the visible light distance Da, which indicates the distance from the visible light imaging device 20 to the first object located within the imaging range of the visible light imaging device 20. In this embodiment, the first visible light camera 21 and the second visible light camera 22 of the visible light imaging device 20 constitute a stereo camera. The visible light distance calculation unit 64 calculates the visible light distance Da from the visible light imaging device 20 to the first object by stereo processing the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22.

[0048] The infrared distance calculation unit 65 calculates the infrared distance Db, which indicates the distance from the infrared imaging device 30 to a second object located within the imaging range of the infrared imaging device 30. In this embodiment, the first infrared camera 31 and the second infrared camera 32 of the infrared imaging device 30 constitute a stereo camera. The infrared distance calculation unit 65 calculates the infrared distance Db from the infrared imaging device 30 to the second object by stereo processing the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32.

[0049] Figure 5 is a schematic diagram illustrating the visible light imaging device 20 and infrared imaging device 30 according to the embodiment. As shown in Figure 5(A), the visible light imaging device 20 images a predetermined area around the work machine 1. As shown in Figure 5(B), the infrared imaging device 30 images a predetermined area around the work machine 1. At least a portion of the imaging range of the visible light imaging device 20 and the imaging range of the infrared imaging device 30 coincide.

[0050] Figure 5 shows a situation in which the visible light image Ga captured by the visible light imaging device 20 does not become blurry. In Figure 5, the first target captured by the visible light imaging device 20 and the second target captured by the infrared imaging device 30 are the same target. The visible light distance calculation unit 64 calculates the visible light distance Da from the visible light imaging device 20 to the first target by stereo processing the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. The infrared distance calculation unit 65 calculates the infrared distance Db from the infrared imaging device 30 to the second target by stereo processing the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32. In Figure 5, the visible light distance Da and the infrared distance Db are substantially equal.

[0051] Figure 6 is a schematic diagram illustrating the visible light image Ga and infrared image Gb according to the embodiment. Figure 6 shows the visible light image Ga and infrared image Gb captured when no phenomenon occurs that causes the visible light image Ga captured by the visible light imaging device 20 to become unclear. In Figure 6, the first object captured by the visible light imaging device 20 and the second object captured by the infrared imaging device 30 are the same object. The first object appearing in the visible light image Ga and the second object appearing in the infrared image Gb are the same object.

[0052] As shown in Figure 6(A), a plurality of first partition regions Pa are defined in the visible light image Ga. Multiple first partition regions Pa are defined in a matrix-like manner in the visible light image Ga. In this embodiment, the first partition region Pa includes pixels (pixel regions) of the visible light image Ga. The pixels of the visible light image Ga correspond to the pixels of the image sensor of the visible light imaging device 20.

[0053] As shown in Figure 6(B), multiple second partition regions Pb are defined in the infrared image Gb. Multiple second partition regions Pb are defined in a matrix-like manner in the infrared image Gb. In this embodiment, the second partition region Pb includes pixels (pixel regions) of the infrared image Gb. The pixels of the infrared image Gb correspond to the pixels of the image sensor of the infrared imaging device 30.

[0054] As shown in Figures 6(A) and 6(B), a reference point Po is set for the image region in both the visible light image Ga and the infrared image Gb. The positions indicated by the reference point Po in both images coincide, and the first partitioned region Pa obtained by dividing the visible light image Ga and the second partitioned region Pb obtained by dividing the infrared image Gb are calibrated so that their size and position (the range indicated by the image) match with respect to the reference point Po.

[0055] In this embodiment, the structure of the image sensor of the visible light imaging device 20 and the structure of the image sensor of the infrared imaging device 30 may be different, as long as the first partitioned region Pa and the second partitioned region Pb of the image are the same. For example, the pixel size of the image sensor of the visible light imaging device 20 and the pixel size of the image sensor of the infrared imaging device 30, the number of pixels (rows and columns) of the image sensor of the visible light imaging device 20 and the number of pixels (rows and columns) of the image sensor of the infrared imaging device 30, and the spacing between adjacent pixels in the image sensor of the visible light imaging device 20 and the spacing between adjacent pixels in the image sensor of the infrared imaging device 30 may be different. Furthermore, the structure of the optical system of the visible light imaging device 20 and the structure of the image sensor of the infrared imaging device 30 may also be different. For example, the focal length of the optical system of the visible light imaging device 20 and the focal length of the optical system of the infrared imaging device 30, and the field of view of the system of the visible light imaging device 20 and the field of view of the optical system of the infrared imaging device 30 may be different.

[0056] Therefore, there is a one-to-one correspondence between the multiple first-order regions Pa (pixels) defined in the visible light image Ga and the multiple second-order regions Pb (pixels) defined in the infrared image Gb, and the regions pointed to by the pixels coincide in position.

[0057] Therefore, the size of the first partitioned region Pa is equal to the size of the second partitioned region Pb. The number of first partitioned regions Pa is equal to the number of second partitioned regions Pb.

[0058] The visible light distance calculation unit 64 calculates the visible light distance Da for each of the multiple first partition regions Pa defined in the visible light image Ga. The visible light distance calculation unit 64 calculates the visible light distance Da from the visible light imaging device 20 to the first object located within the imaging range of the visible light imaging device 20 for each of the multiple first partition regions Pa of the visible light image Ga.

[0059] The infrared distance calculation unit 65 calculates the infrared distance Db for each of the multiple second partition regions Pb defined in the infrared image Gb. The infrared distance calculation unit 65 calculates the infrared distance Db from the infrared imaging device 30 to the second object located within the imaging range of the infrared imaging device 30 for each of the multiple second partition regions Pb of the infrared image Gb.

[0060] The determination unit 66 determines whether the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a predetermined distance threshold for each of the corresponding first partition region Pa and second partition region Pb.

[0061] Figure 7 is a schematic diagram illustrating the visible light imaging device 20 and infrared imaging device 30 according to the embodiment. Figure 7 shows a situation in which the visible light image Ga captured by the visible light imaging device 20 becomes unclear.

[0062] During operation of machine 1, there is a possibility that the visible light image Ga may become unclear. As shown in Figure 7, dust generation caused by the operation of machine 1 is an example of an event that may cause the visible light image Ga to become unclear. As shown in Figure 7(A), at least a portion of the dust may be generated in the space between the visible light imaging device 20 and the object being imaged. As shown in Figure 7(B), at least a portion of the dust may be generated in the space between the infrared imaging device 30 and the object being imaged.

[0063] Visible light cannot penetrate dust, and therefore the visible light imaging device 20 cannot image objects that are obscured by dust. The first object imaged by the visible light imaging device 20 includes dust and a part of the object being imaged.

[0064] Infrared light can penetrate dust, and the infrared imaging device 30 can image objects that are obscured by dust. The second object imaged by the infrared imaging device 30 contains almost no dust and includes the object being imaged.

[0065] As described above, the visible light distance calculation unit 64 calculates the visible light distance Da from the visible light imaging device 20 to the first target by stereo processing the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. The visible light distance Da calculated by the visible light distance calculation unit 64 includes the visible light distance Da from the visible light imaging device 20 to the dust and the visible light distance Da from the visible light imaging device 20 to the target being imaged. The visible light distance Da from the visible light imaging device 20 to the dust is shorter than the visible light distance Da from the visible light imaging device 20 to the target being imaged.

[0066] As described above, the infrared distance calculation unit 65 calculates the infrared distance Db from the infrared imaging device 30 to the second target by stereo processing the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32. The infrared distance Db calculated by the visible light distance calculation unit 64 does not include the infrared distance Db from the infrared imaging device 30 to the dust, but it does include the infrared distance Db from the infrared imaging device 30 to the target being imaged. The visible light distance Da from the visible light imaging device 20 to the dust is shorter than the infrared distance Db from the infrared imaging device 30 to the target being imaged.

[0067] Thus, if dust is present in the space between the visible light imaging device 20 and the infrared imaging device 30 and the object to be imaged, a difference may occur between the visible light distance Da and the infrared distance Db. The determination unit 66 can determine the presence or absence of dust by determining whether the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold. Furthermore, the determination unit 66 can identify the location of the dust by determining whether the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold for each of the corresponding first and second compartment regions Pa and Pb.

[0068] Figure 8 is a schematic diagram illustrating the visible light image Ga and infrared image Gb according to the embodiment. Figure 8 shows the visible light image Ga and infrared image Gb captured when an event occurs in which the visible light image Ga captured by the visible light imaging device 20 becomes unclear. Figure 8(A) shows the visible light image Ga captured in a situation where dust is generated as described with reference to Figure 7(A). Figure 8(B) shows the infrared image Gb captured in a situation where dust is generated as described with reference to Figure 7(B).

[0069] As shown in Figure 8, dust can cause the first object imaged by the visible light imaging device 20 and the second object imaged by the infrared imaging device 30 to no longer coincide. The first object imaged by the visible light imaging device 20 includes dust and a portion of the object being imaged. The second object imaged by the infrared imaging device 30 includes the object being imaged, but contains almost no dust.

[0070] As shown in Figure 8(A), the visible light image Ga may not show a portion of the object being imaged that is obscured by dust. The visible light image Ga includes both the dust and a portion of the object being imaged.

[0071] As shown in Figure 8(B), the infrared image Gb does not show dust particles, but it does show the object being imaged. The infrared image Gb contains almost no dust particles and includes the object being imaged.

[0072] The synthesis unit 67 synthesizes the second partition region Pb, where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold, with the visible light image Ga to generate a synthesized image Gd.

[0073] In this embodiment, the second compartment region Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to the distance threshold is a plurality of second compartment regions Pb included in the cropped region Gc of Figure 8(B). The second compartment region Pb of the cropped region Gc corresponds to the first compartment region Pa where dust appears in the visible light image Ga.

[0074] The second partition region Pb of the cut-out region Gc is a second partition region Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to the distance threshold. In the embodiment, the cut-out region Gc represents a collection of multiple second partition regions Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to the distance threshold. The second partition region Pb outside the cut-out region Gc is a second partition region Pb where the difference between the visible light distance Da and the infrared distance Db is less than the distance threshold.

[0075] Figures 9 and 10 are schematic diagrams illustrating the method for generating the composite image Gd according to the embodiment.

[0076] As shown in Figure 9(B), the synthesis unit 67 cuts out a cutout region Gc from the infrared image Gb. That is, the synthesis unit 67 cuts out a plurality of second partition regions Pb from the infrared image Gb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold.

[0077] As shown in Figure 9(A), the synthesis unit 67 removes a plurality of first partition regions Pa corresponding to the cropped region Gc from the visible light image Ga. That is, the synthesis unit 67 removes a plurality of first partition regions Pa from the visible light image Ga where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold.

[0078] A first partition region Pa corresponding to the cut-out region Gc is a first partition region Pa where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to the distance threshold. A first partition region Pa not corresponding to the cut-out region Gc is a first partition region Pa where the difference between the visible light distance Da and the infrared distance Db is less than the distance threshold.

[0079] As shown in Figure 10, the synthesis unit 67 synthesizes a cropped region Gc cut out from an infrared image Gb and a visible light image Ga from which a plurality of first partition regions Pa corresponding to the cropped region Gc have been removed to generate a composite image Gd. The synthesis unit 67 synthesizes the cropped region Gc and the visible light image Ga such that the cropped region Gc, which includes a plurality of second partition regions Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold, is superimposed on the first partition region Pa of the visible light image Ga corresponding to the cropped region Gc. In this embodiment, the synthesis unit 67 fits the cropped region Gc, which includes a plurality of second partition regions Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold, into the portion of the visible light image Ga from which the first partition region Pa has been removed.

[0080] Furthermore, after the composite unit 67 has fitted (superimposed) the cropped region Gc onto the visible light image Ga, it may also smooth the boundary between the visible light image Ga and the cropped region Gc.

[0081] As shown in Figure 10, the composite image Gd includes a plurality of second-order regions Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold, and a plurality of first-order regions Pa where the difference between the visible light distance Da and the infrared distance Db is less than a distance threshold. In other words, the composite image Gd is composed of a portion of the visible light image Ga and a portion of the infrared image Gb.

[0082] The guideline generation unit 68 generates a guideline image Ge indicating the distance from the work machine 1 based on the infrared distance Db calculated by the infrared distance calculation unit 65.

[0083] The display output unit 69 outputs the composite image Gd generated by the synthesis unit 67 to the display device 50 so that the composite image Gd is displayed on the display device 50. The display output unit 69 also outputs the guideline image Ge generated by the guideline generation unit 68 to the display device 50 so that the guideline image Ge is displayed on the display device 50.

[0084] Figure 11 is a schematic diagram showing the composite image Gd and the guideline image Ge displayed in the display device 50 according to the embodiment.

[0085] When the composite image Gd is displayed on the display device 50, even if dust is generated, the operator of the work machine 1 can check the display device 50 and recognize the situation around the work machine 1. In addition, when the guideline image Ge is displayed on the display device 50, the operator of the work machine 1 can check the display device 50 and recognize the distance from the work machine 1 to the image target (construction target).

[0086] [Display method] Figure 12 is a flowchart showing the display method according to the embodiment.

[0087] The visible light imaging device 20 images the area around the work machine 1. The infrared imaging device 30 images the area around the work machine 1. The image output unit 304 transmits visible light image data, showing the visible light image Ga captured by the visible light imaging device 20, to the control device 60 via the communication device 7 and the communication system 400. The image output unit 304 also transmits infrared image data, showing the infrared image Gb captured by the infrared imaging device 30, to the control device 60 via the communication device 7 and the communication system 400.

[0088] The visible light image acquisition unit 62 acquires the visible light image Ga transmitted from the image output unit 304 (step S1).

[0089] The infrared image acquisition unit 63 acquires the infrared image Gb transmitted from the image output unit 304 (step S2).

[0090] The visible light distance calculation unit 64 calculates the visible light distance Da, which indicates the distance from the visible light imaging device 20 to the first target, for each of the multiple first partition regions Pa defined in the visible light image Ga by stereo processing the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22 (step S3).

[0091] The infrared distance calculation unit 65 performs stereo processing on the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32 to calculate the infrared distance Db, which indicates the distance from the infrared imaging device 30 to the second object, for each of the multiple second section regions Pb defined in the infrared image Gb to correspond to the first section region Pa (step S4).

[0092] The determination unit 66 determines whether the difference between the visible light distance Da and the infrared distance Db for each corresponding first partition region Pa and second partition region Pb is greater than or equal to a predetermined distance threshold. That is, the determination unit 66 determines whether the difference between the visible light distance Da and the infrared distance Db is large for each corresponding first partition region Pa and second partition region Pb. The determination unit 66 also determines whether there is a second partition region Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to the distance threshold (step S5).

[0093] In step S5, if it is determined that there exists a second partition region Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold, that is, if it is determined that the difference between the visible light distance Da and the infrared distance Db is large (step S5: Yes), the determination unit 66 determines whether the cropped region Gc, which represents a collection of multiple second partition regions Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold, is greater than or equal to a predetermined region threshold. That is, the determination unit 66 determines whether the size (proportion) of the cropped region Gc occupying the infrared image Gb is large (step S6).

[0094] In step S6, if it is determined that the cropped area Gc is greater than or equal to the area threshold (step S6: Yes), that is, if it is determined that the size of the cropped area Gc occupying the infrared image Gb is large, the synthesis unit 67 synthesizes the cropped area Gc and the visible light image Ga to generate a synthesized image Gd (step S7).

[0095] Furthermore, the guideline generation unit 68 generates a guideline image Ge indicating the distance from the work machine 1 based on the infrared distance Db calculated by the infrared distance calculation unit 65.

[0096] The display output unit 69 outputs the composite image Gd generated by the synthesis unit 67 to the display device 50 so that the composite image Gd is displayed on the display device 50. The display output unit 69 also outputs the guideline image Ge generated by the guideline generation unit 68 to the display device 50 so that the guideline image Ge is displayed on the display device 50. As a result, the composite image Gd and the guideline image Ge are displayed on the display device 50 as shown in Figure 11 (step S8).

[0097] In step S5, if it is determined that there is no second partition region Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to the distance threshold (step S5: No), the display output unit 69 outputs the visible light image Ga to the display device 50 so that the visible light image Ga is displayed on the display device 50. The display output unit 69 also outputs the guideline image Ge generated by the guideline generation unit 68 to the display device 50 so that the guideline image Ge is displayed on the display device 50. As a result, the visible light image Ga and the guideline image Ge are displayed on the display device 50 (step S9).

[0098] In step S6, if it is determined that the cropping region Gc is less than the region threshold, that is, if it is determined that the size of the cropping region Gc occupying the infrared image Gb is small (step S6: Yes), the display output unit 69 outputs the visible light image Ga to the display device 50 so that the visible light image Ga is displayed on the display device 50. The display output unit 69 also outputs the guideline image Ge generated by the guideline generation unit 68 to the display device 50 so that the guideline image Ge is displayed on the display device 50. As a result, the visible light image Ga and the guideline image Ge are displayed on the display device 50 (step S9).

[0099] In other words, in the embodiment, even if there is a second partitioned area Pb where the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold, if the number of such second partitioned areas Pb is small, that is, if the cropping area Gc is small, the composite image Gd is not displayed on the display device 50, and the visible light image Ga is displayed on the display device 50. In other words, even if dust is generated, if the area of ​​the imaging target that is obscured by the dust is small, the composite image Gd is not displayed on the display device 50, and the visible light image Ga is displayed on the display device 50.

[0100] Note that the process in step S6 is optional.

[0101] [Computer System] Figure 13 is a block diagram showing a computer system 1000 according to an embodiment. The control device 60 described above includes the computer system 1000. The computer system 1000 has a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 1003, and an interface 1004 including input / output circuits. The functions of the control device 60 described above are stored in the storage 1003 as a computer program. The processor 1001 reads the computer program from the storage 1003, loads it into the main memory 1002, and executes the above-described processing according to the program. The computer program may be distributed to the computer system 1000 via a network.

[0102] The computer program or computer system 1000 can perform the following actions in accordance with the above-described embodiment: acquire a visible light image Ga showing an image of a first target captured by a visible light imaging device 20 that images a predetermined range around the work machine 1; acquire an infrared image Gb showing an image of a second target captured by an infrared imaging device 30 that images a predetermined range around the work machine 1; calculate a visible light distance Da indicating the distance from the visible light imaging device 20 to the first target for each of the multiple first partition regions Pa defined in the visible light image Ga; calculate an infrared distance Db indicating the distance from the infrared imaging device 30 to the second target for each of the multiple second partition regions Pb defined in the infrared image Gb to correspond to the first partition region Pa; determine whether the difference between the visible light distance Da and the infrared distance Db for each corresponding first partition region Pa and second partition region Pb is greater than or equal to a distance threshold; synthesize the second partition region Pb whose difference is greater than or equal to the distance threshold with the visible light image Ga to generate a composite image Gd; and output the composite image Gd so that it is displayed on the display device 50.

[0103] [effect] As explained above, in situations where no event causing the visible light image Ga to become unclear occurs, the visible light image Ga is displayed on the display device 50 as described in step S9. The visible light image Ga is a color image, and its resolution is higher than that of the infrared image Gb. In other words, the visibility of the visible light image Ga is superior to that of the infrared image Gb. Therefore, in situations where no event causing the visible light image Ga to become unclear occurs, the display system 10 displays the visible light image Ga on the display device 50. This allows the display system 10 to provide the operator of the work machine 1 with information about the surrounding environment of the work machine 1.

[0104] In situations where the visible light image Ga becomes unclear, a composite image Gd of the visible light image Ga and the infrared image Gb is displayed on the display device 50, as described in step S8. Only a portion of the unclear area in the visible light image Ga is replaced by the infrared image Gb. As mentioned above, the visibility of the visible light image Ga is excellent. Since only a portion of the unclear area in the visible light image Ga is replaced by the infrared image Gb, rather than the entire area of ​​the visible light image Ga being replaced by the infrared image Gb, the visibility of the composite image Gd is maintained at a good level.

[0105] The areas of the visible light image Ga that are replaced by the infrared image Gb are determined based on the difference between the visible light distance Da and the infrared distance Db. If there are areas in both the visible light image Ga and the infrared image Gb where the difference between the visible light distance Da and the infrared distance Db is large, the visible light image Ga and the infrared image Gb are combined. As a result, only the unclear areas of the visible light image Ga are properly replaced by the infrared image Gb. The display system 10 can provide the operator of the work machine 1 with information about the surrounding conditions of the work machine 1.

[0106] [Other embodiments] In the above-described embodiment, the phenomenon causing the image captured by the visible light imaging device 20 to become unclear was assumed to be the generation of dust. Other examples of phenomena causing the image captured by the visible light imaging device 20 to become unclear include the generation of fog, insufficient visible light due to the operation of the work machinery at night, and the subject being backlit.

[0107] When fog particles (water droplets) are present in the space between the visible light imaging device 20 and the infrared imaging device 30 and the object to be imaged, visible light has difficulty penetrating the fog particles, but infrared light can penetrate them. Therefore, the display system 10 can generate a composite image Gd according to the above-described embodiment. Even if at least a portion of the visible light image Ga captured by the visible light imaging device 20 becomes unclear, the display system 10 can provide the operator of the work machine 1 with information about the surrounding conditions of the work machine 1 by displaying the composite image Gd on the display device 50.

[0108] When the amount of visible light is insufficient, the brightness of the pixels in the image sensor of the visible light imaging device 20 becomes insufficient. In other words, so-called black crushing occurs. In that case, it becomes difficult for the visible light distance calculation unit 64 to stereo-process the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. That is, when the amount of visible light is insufficient, it becomes difficult for the visible light distance calculation unit 64 to calculate the visible light distance Da. On the other hand, the infrared distance calculation unit 65 can calculate the infrared distance Db by stereo-processing the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32. Even when the amount of visible light is insufficient, a difference occurs between the visible light distance Da and the infrared distance Db, so the display system 10 can generate a composite image Gd according to the embodiment described above.

[0109] When the object to be imaged is in a backlit state, the brightness of the pixels in the image sensor of the visible light imaging device 20 becomes excessive. In other words, so-called overexposure occurs. In that case, it becomes difficult for the visible light distance calculation unit 64 to stereo-process the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. That is, when the object to be imaged is in a backlit state, it becomes difficult for the visible light distance calculation unit 64 to calculate the visible light distance Da. On the other hand, the infrared distance calculation unit 65 can calculate the infrared distance Db by stereo-processing the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32. Even when the object to be imaged is in a backlit state, a difference occurs between the visible light distance Da and the infrared distance Db, so the display system 10 can generate a composite image Gd according to the above embodiment.

[0110] In the above embodiment, the visible light distance calculation unit 64 calculates the visible light distance Da by stereo processing the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. The work machine 1 may be equipped with a distance detection device such as a laser sensor (LIDAR: Light Detection and Ranging). The laser sensor detects distance using laser light, which is within the wavelength range of visible light. Based on the detection data from the distance detection device, the visible light distance calculation unit 64 can calculate the visible light distance Da from the visible light imaging device 20 to the first target for each of the multiple first partition regions Pa defined in the visible light image Ga. If a distance detection device is provided, one of the first visible light camera 21 and the second visible light camera 22 of the visible light imaging device 20 may be omitted.

[0111] In the above embodiment, one first partitioned region Pa may be a single pixel of the visible light image Ga, or a combination of multiple pixels. One second partitioned region Pb may be a single pixel of the infrared image Gb, or a combination of multiple pixels. Furthermore, the first partitioned region Pa may be defined without being based on pixels of the visible light image Ga. The second partitioned region Pb may also be defined without being based on pixels of the infrared image Gb. It is sufficient that there is a one-to-one correspondence between multiple first partitioned regions Pa and multiple second partitioned regions Pb.

[0112] In the above-described embodiment, the display system 10 is applied to the remote control system 100. The display device 50 does not have to be located in the remote control room 200. The display device 50 may be located in the control room (cab) of the work machine 1. In addition, some functions of the control device 60 described in the above-described embodiment may be located in the work machine 1. An operator who is in the control room of the work machine 1 can operate the boarding control device located in the control room of the work machine 1 while checking the display device 50 located in the control room of the work machine 1. Even in this case, the display system 10 can provide the operator of the work machine 1 with information about the surroundings of the work machine 1, even if the visible light image Ga captured by the visible light imaging device 20 becomes unclear.

[0113] In the above-described embodiment, the work machine 1 is assumed to be a hydraulic excavator. The work machine 1 may also be a bulldozer, a wheel loader, or a dump truck. [Explanation of symbols]

[0114] 1...Work machine, 2...Traveling body, 3...Slewing body, 4...Work machine, 4A...Boom, 4B...Arm, 4C...Bucket, 5...Hydraulic cylinder, 5A...Boom cylinder, 5B...Arm cylinder, 5C...Bucket cylinder, 6...Communication device, 7...Communication device, 10...Display system, 20...Visible light imaging device, 21...First visible light camera, 22...Second visible light camera, 30...Infrared imaging device, 31...First infrared camera, 32...Second infrared camera, 40...Remote control device, 45...Operator's seat, 50...Display device, 60...Control device, 61...Operation signal output unit, 62...Visible light image acquisition unit, 6 3...Infrared image acquisition unit, 64...Visible light distance calculation unit, 65...Infrared distance calculation unit, 66...Determination unit, 67...Composite unit, 68...Guideline generation unit, 69...Display output unit, 100...Remote control system, 200...Remote control room, 300...Control device, 301...Traveling unit control unit, 302...Slewing unit control unit, 303...Work machine control unit, 304...Image output unit, 400...Communication system, Da...Visible light distance, Db...Infrared distance, Ga...Visible light image, Gb...Infrared image, Gc...Cropped area, Gd...Composite image, Ge...Guideline image, Pa...First section area, Pb...Second section area, RX...Slewing axis.

Claims

1. A visible light image acquisition unit acquires a visible light image showing the target being imaged, which is captured by a visible light imaging device installed on the work machine. An infrared image acquisition unit acquires an infrared image showing the target being imaged, which is captured by an infrared imaging device provided on the work machine, simultaneously with the acquisition of the visible light image. A visible light distance calculation unit calculates a visible light distance, which indicates the distance from the visible light imaging device to the object to be imaged, for each of the multiple first partitioned regions defined in the visible light image. An infrared distance calculation unit calculates an infrared distance indicating the distance from the infrared imaging device to the object to be imaged for each of a plurality of second partition regions defined in the infrared image to correspond to the first partition region, If the difference between the visible light distance and the infrared distance is greater than or equal to a distance threshold, a combining unit generates a composite image by combining the infrared image and the visible light image. The system includes a display output unit that outputs the composite image so that it is displayed on a display device, The synthesis unit synthesizes the second partition region and the visible light image such that the second partition region, whose difference is greater than or equal to a distance threshold, is superimposed on the first partition region of the visible light image corresponding to the second partition region. Display system.

2. The composite image includes a second partitioned region where the difference is greater than or equal to a distance threshold and a first partitioned region where the difference is less than a distance threshold. The display system according to claim 1.

3. The visible light imaging device includes a first visible light camera and a second visible light camera. The visible light distance calculation unit calculates the visible light distance by stereo processing the visible light image captured by the first visible light camera and the visible light image captured by the second visible light camera. The display system according to claim 1 or claim 2.

4. The infrared imaging device includes a first infrared camera and a second infrared camera. The infrared distance calculation unit calculates the infrared distance by stereo processing the infrared image captured by the first infrared camera and the infrared image captured by the second infrared camera. The display system according to any one of claims 1 to 3.

5. The system includes a guideline generation unit that generates a guideline image indicating the distance from the work machine based on the infrared distance, The display output unit outputs the guideline image so that the guideline image is displayed on the display device. The display system according to any one of claims 1 to 4.

6. To acquire a visible light image showing the target being imaged by a visible light imaging device installed on the work machine, Simultaneously with the acquisition of the visible light image, an infrared image showing the target being imaged is acquired by an infrared imaging device provided on the work machine. The visible light distance, which indicates the distance from the visible light imaging device to the object to be imaged, is calculated for each of the multiple first partition regions defined in the visible light image. The infrared distance, which indicates the distance from the infrared imaging device to the object to be imaged, is calculated for each of the multiple second partition regions defined in the infrared image to correspond to the first partition region. If the difference between the visible light distance and the infrared distance is greater than or equal to a distance threshold, the infrared image and the visible light image are combined to generate a composite image. This includes outputting the composite image so that it is displayed on a display device, The second partitioned region and the visible light image are combined such that the second partitioned region, whose difference is greater than or equal to a distance threshold, is superimposed on the first partitioned region of the visible light image corresponding to the second partitioned region. Display method.

Images