Display system and display method

Abstract

To provide an operator of a work machine with information on the situation around the work machine even when an event occurs in which an image captured by a visible light imaging device becomes unclear.SOLUTION: A display system includes a visible light image acquisition unit for acquiring a visible light image of a first object captured by a visible light imaging device, an infrared image acquisition unit for acquiring an infrared image of a second object captured by an infrared imaging device, a visible light distance calculation unit for calculating a visible light distance indicating the distance from the visible light imaging device to the first object for each of a plurality of first sectional areas, an infrared distance calculation unit for calculating an infrared distance indicating the distance from the infrared imaging device to the second object for each of a plurality of second sectional areas, a determination unit for determining whether the difference between the visible light distance and the infrared distance for each of the corresponding first and second divided areas is equal to or greater than a distance threshold value, a combination unit for combining the second sectional area where the difference is equal to or greater than the distance threshold value with the visible light image to generate a composite image, and a display output unit for outputting the composite image such that the composite image is displayed on a display device.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a display system and a display method.

Background Art

[0002] In the technical field related to display systems, an image processing apparatus as disclosed in Patent Document 1 is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technical field related to work machines, a technique of imaging the periphery of a work machine with a visible light imaging device is known. By providing an image captured by the visible light imaging device to an operator of the work machine, the operator can check the situation around the work machine. Dust may be generated due to the operation of the work machine. In addition, the operation of the work machine may be carried out in a situation where fog is present. When dust or fog is generated, the image captured by the visible light imaging device may become unclear. Also, even when the operation of the work machine is carried out at night, the image captured by the visible light imaging device may become unclear. Further, even when imaging an imaging target in a backlight state, the image captured by the visible light imaging device may become unclear.

[0005] An object of the present disclosure is to provide the situation around a work machine to an operator of the work machine even when an event occurs in which an image captured by a visible light imaging device becomes unclear.

Means for Solving the Problems

[0006] According to the present disclosure, a visible light image acquisition unit that acquires a visible light image showing an image of a first object captured by a visible light imaging device provided on a work machine, an infrared image acquisition unit that acquires an infrared image showing an image of a second object captured by an infrared imaging device provided on the work machine, a visible light distance calculation unit that calculates a visible light distance indicating the distance from the visible light imaging device to the first object for each of a plurality of first section regions defined in the visible light image, an infrared distance calculation unit that calculates an infrared distance indicating the distance from the infrared imaging device to the second object for each of a plurality of second section regions defined in the infrared image so as to correspond to the first section region, a determination unit that determines whether or not a difference between the visible light distance and the infrared distance is equal to or greater than a distance threshold for each corresponding first section region and second section region, a synthesis unit that synthesizes the second section region in which the difference is equal to or greater than the distance threshold and the visible light image to generate a synthesized image, and a display output unit that outputs the synthesized image so that the synthesized image is displayed on a display device. A display system is provided.

Advantages of the Invention

[0007] According to the present disclosure, even when an event occurs in which an image captured by a visible light imaging device becomes unclear, the situation around the work machine can be provided to an operator of the work machine.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, but the present disclosure is not limited to the embodiments. The components of the embodiments described below can be combined as appropriate. Also, some components may not be used.

[0010] [Remote Operation System] FIG. 1 is a schematic diagram showing a remote operation system 100 of a working machine 1 according to an embodiment. The remote operation system 100 remotely operates the working machine 1 existing at the work site. At least a part of the remote operation system 100 is arranged in a remote operation room 200 at the remote operation location. The remote operation system 100 includes a remote operation device 40, a display device 50, and a control device 60.

[0011] The remote operation device 40 is arranged in the remote operation room 200 outside the working machine 1. The remote operation device 40 is operated by an operator in the remote operation room 200. The operator can operate the remote operation device 40 while sitting on the operator's seat 45.

[0012] The display device 50 is arranged in the remote operation room 200 outside the working machine 1. The display device 50 displays an image of the work site. The image of the work site includes an image of a predetermined range around the working machine 1. The image of the predetermined range around the working machine 1 includes at least an image of the work target of the working machine 1. The work target of the working machine 1 includes the construction target of the working machine 1.

[0013] The display device 50 includes a flat panel display such as a liquid crystal display (LCD) or an organic EL display (OELD). In an embodiment, the display device 50 includes a plurality of flat panel displays arranged adjacent to each other. The display device 50 may be composed of one flat panel display.

[0014] The operator operates the remote operation device 40 while checking the image of the work site displayed on the display device 50. The working machine 1 is remotely operated by the remote operation device 40.

[0015] The control device 60 is arranged in the remote operation room 200 outside the working machine 1. The control device 60 includes a computer system.

[0016] The working machine 1 is provided with a control device 300. The control device 300 includes a computer system.

[0017] The control device 60 and the control device 300 communicate with each other via the communication system 400. Examples of the communication system 400 include the Internet, a local area network (LAN), a mobile phone communication network, and a satellite communication network.

[0018] [Construction machine] FIG. 2 is a perspective view showing a construction machine 1 according to an embodiment. In the embodiment, the construction machine 1 is assumed to be a hydraulic excavator. The construction machine 1 operates at a work site.

[0019] As shown in FIG. 2, the construction machine 1 includes a traveling body 2, a revolving body 3 supported by the traveling body 2, a working machine 4 supported by the revolving body 3, a hydraulic cylinder 5 for driving the working machine 4, a visible light imaging device 20, and an infrared imaging device 30.

[0020] The traveling body 2 can travel while supporting the revolving body 3. The revolving body 3 can revolve about a revolving axis RX while being supported by the traveling body 2. The working machine 4 includes a boom 4A connected to the revolving body 3, an arm 4B connected to the boom 4A, a bucket 4C connected to the arm 4B, and, as the hydraulic cylinder 5, a boom cylinder 5A for driving the boom 4A, an arm cylinder 5B for driving the arm 4B, and a bucket cylinder 5C for driving the bucket 4C.

[0021] The direction in which the working machine 4 exists with reference to the revolving axis RX is the front, and the opposite direction of the front is the rear. One of the left and right directions with reference to the revolving axis RX is the right, and the opposite direction of the right is the left. The direction away from the ground contact surface of the traveling body 2 is the upward, and the opposite direction of the upward is the downward.

[0022] The visible light imaging device 20 images the work site. The visible light imaging device 20 is provided on the construction machine 1. In the embodiment, the visible light imaging device 20 is provided on the revolving body 3. The visible light imaging device 20 images a predetermined range around the construction machine 1. The visible light imaging device 20 acquires an image in the wavelength range of visible light. The wavelength range of visible light is, for example, 360 [nm] or more and 830 [nm] or less.

[0023] The infrared imaging device 30 images the work site. The infrared imaging device 30 is provided on the work machine 1. In the embodiment, the infrared imaging device 30 is provided on the revolving body 3. The infrared imaging device 30 images a predetermined range around the work machine 1. The infrared imaging device 30 acquires an image in the spectral range of infrared rays. The spectral range of the infrared rays is 780 [nm] or more and 100 [μm] or less. In the embodiment, the infrared imaging device 30 acquires an image in the spectral range of far infrared rays. The spectral range of the infrared imaging device 30 is, for example, 7.5 [μm] or more and 14 [μm] or less.

[0024] Each of the visible light imaging device 20 and the infrared imaging device 30 images an imaging target existing around the work machine 1. The imaging target is an object. Examples of the imaging target imaged by the visible light imaging device 20 and the infrared imaging device 30 include a construction target of the work machine 1, an excavation target of the work machine 4, a structure existing at the work site, at least a part of the work machine 1, a work machine different from the work machine 1, and a person (worker) working at the work site.

[0025] Each of the visible light imaging device 20 and the infrared imaging device 30 includes an optical system and an image sensor that receives light that has passed through the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0026] At least a part of the imaging range of the visible light imaging device 20 coincides with at least a part of the imaging range of the infrared imaging device 30. The imaging range of the visible light imaging device 20 includes the field of view range of the optical system of the visible light imaging device 20. The imaging range of the infrared imaging device 30 includes the field of view range of the optical system of the infrared imaging device 30. In the embodiment, the imaging range of the visible light imaging device 20 coincides with the imaging range of the infrared imaging device 30. Note that a part of the imaging range of the visible light imaging device 20 may coincide with a part of the imaging range of the infrared imaging device 30.

[0027] [Visible Light Imaging Device and Far-Infrared Imaging Device] FIG. 3 is a perspective view showing a visible light imaging device 20 and an infrared imaging device 30 according to an embodiment. As shown in FIG. 3, each of the visible light imaging device 20 and the infrared imaging device 30 is disposed above the upper part of the front portion of the revolving body 3. Each of the visible light imaging device 20 and the infrared imaging device 30 images the front of the revolving body 3. In the embodiment, a predetermined range around the working machine 1 imaged by the visible light imaging device 20 and the infrared imaging device 30 is the front range of the revolving body 3.

[0028] In the embodiment, the visible light imaging device 20 includes a first visible light camera 21 and a second visible light camera 22. Each of the first visible light camera 21 and the second visible light camera 22 acquires an image in the wavelength range of visible light. The first visible light camera 21 and the second visible light camera 22 are arranged in the left-right direction.

[0029] In the embodiment, the infrared imaging device 30 includes a first infrared camera 31 and a second infrared camera 32. Each of the first infrared camera 31 and the second infrared camera 32 acquires an image in the spectral range of infrared rays. In the embodiment, each of the first infrared camera 31 and the second infrared camera 32 is a far-infrared camera. The first infrared camera 31 and the second infrared camera 32 are arranged in the left-right direction.

[0030] The first infrared camera 31 is disposed to the left of the first visible light camera 21. The second infrared camera 32 is disposed to the right of the second visible light camera 22. The first visible light camera 21 and the second visible light camera 22 are disposed between the first infrared camera 31 and the second infrared camera 32.

[0031] Note that the first infrared camera 31 and the second infrared camera 32 may be disposed between the first visible light camera 21 and the second visible light camera 22.

[0032] The imaging ranges of the first visible light camera 21, the imaging range of the second visible light camera 22, the imaging range of the first infrared camera 31, and the imaging range of the second infrared camera 32 coincide. The first visible light camera 21, the second visible light camera 22, the first infrared camera 31, and the second infrared camera 32 simultaneously image the front range of the rotating body 3.

[0033] It should be noted that it is sufficient if at least a part of the imaging range of the first visible light camera 21 coincides with the imaging range of the second visible light camera 22. It is sufficient if at least a part of the imaging range of the first infrared camera 31 coincides with the imaging range of the second infrared camera 32. It is sufficient if a part of the imaging range of the visible light imaging device 20 coincides with the imaging range of the infrared imaging device 30.

[0034] In the following description, the image captured by the visible light imaging device 20 is appropriately referred to as the visible light image Ga, and the image captured by the infrared imaging device 30 is appropriately referred to as the infrared image Gb.

[0035] [Display System] FIG. 4 is a functional block diagram showing the remote operation system 100 of the working machine 1 according to the embodiment. As shown in FIG. 4, the remote operation system 100 includes a display system 10 for displaying an image of the work site. The remote operation system 100 also includes a communication device 6 disposed at the remote operation location, a control device 60 connected to the communication device 6, a remote operation device 40 connected to the control device 60, and a display device 50 connected to the control device 60. The remote operation system 100 further includes a communication device 7 disposed on the working machine 1, a control device 300 connected to the communication device 7, a visible light imaging device 20 connected to the control device 300, an infrared imaging device 30 connected to the control device 300, a traveling body 2 controlled by the control device 300, a rotating body 3 controlled by the control device 300, and a hydraulic cylinder 5 controlled by the control device 300. The display system 10 includes the visible light imaging device 20, the infrared imaging device 30, the control device 60, and the display device 50.

[0036] The control device 300 includes a traveling control unit 301, a turning control unit 302, a work implement control unit 303, and an image output unit 304.

[0037] The traveling control unit 301 receives the operation signal of the remote control device 40 transmitted from the control device 60. The traveling control unit 301 outputs a control signal for controlling the operation of the traveling body 2 based on the operation signal of the remote control device 40.

[0038] The turning control unit 302 receives the operation signal of the remote control device 40 transmitted from the control device 60. The turning control unit 302 outputs a control signal for controlling the operation of the turning body 3 based on the operation signal of the remote control device 40.

[0039] The work implement control unit 303 receives the operation signal of the remote control device 40 transmitted from the control device 60. The work implement control unit 303 outputs a control signal for controlling the operation of the work implement 4 based on the operation signal of the remote control device 40. The control signal for controlling the work implement 4 includes a control signal for controlling the hydraulic cylinder 5.

[0040] The image output unit 304 outputs visible light image data indicating the visible light image Ga captured by the visible light imaging device 20. Further, the image output unit 304 outputs infrared image data indicating the infrared image Gb captured by the infrared imaging device 30.

[0041] The communication device 7 communicates with the communication device 6 via the communication system 400. The communication device 7 receives the operation signal of the remote control device 40 transmitted from the control device 60 via the communication device 6 and outputs it to the control device 300. The communication device 7 transmits the visible light image data and the infrared image data output from the image output unit 304 to the communication device 6. The communication device 7 includes an encoder that compresses each of the visible light image data and the infrared image data. Each of the visible light image data and the infrared image data is transmitted from the communication device 7 to the communication device 6 in a compressed state.

[0042] The communication device 6 communicates with the communication device 7 via the communication system 400. The communication device 6 transmits an operation signal generated by operating the remote control device 40 to the communication device 7. The communication device 6 receives the visible light image data and the infrared image data transmitted from the control device 300 via the communication device 7 and outputs them to the control device 60. The communication device 6 includes a decoder that restores each of the compressed visible light image data and infrared image data. Each of the visible light image data and the infrared image data is output from the communication device 6 to the control device 60 in a restored state.

[0043] The control device 60 includes an operation signal output unit 61, a visible light image acquisition unit 62, an infrared image acquisition unit 63, a visible light distance calculation unit 64, an infrared distance calculation unit 65, a determination unit 66, a synthesis unit 67, a reference line generation unit 68, and a display output unit 69.

[0044] The operation signal output unit 61 outputs an operation signal for remotely operating the working machine 1. When the remote control device 40 is operated by an operator, an operation signal for remotely operating the working machine 1 is generated. The operation signal output unit 61 outputs the operation signal of the remote control device 40. The communication device 6 transmits the operation signal output from the operation signal output unit 61 to the communication device 7.

[0045] The visible light image acquisition unit 62 acquires a visible light image Ga indicating an image of a first target imaged by the visible light imaging device 20. The visible light image acquisition unit 62 acquires the visible light image Ga by acquiring the visible light image data restored by the communication device 6. The first target includes an imaging target existing within the imaging range of the visible light imaging device 20.

[0046] The infrared image acquisition unit 63 acquires an infrared image Gb indicating an image of a second target imaged by the infrared imaging device 30. The infrared imaging device 30 acquires the infrared image Gb by acquiring the infrared image data restored by the communication device 6. The second target includes an imaging target existing within the imaging range of the infrared imaging device 30.

[0047] The visible light distance calculation unit 64 calculates a visible light distance Da indicating the distance from the visible light imaging device 20 to a first target existing in the imaging range of the visible light imaging device 20. In the embodiment, the first visible light camera 21 and the second visible light camera 22 of the visible light imaging device 20 constitute a stereo camera. The visible light distance calculation unit 64 calculates the visible light distance Da from the visible light imaging device 20 to the first target by performing stereo processing on the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22.

[0048] The infrared distance calculation unit 65 calculates an infrared distance Db indicating the distance from the infrared imaging device 30 to a second target existing in the imaging range of the infrared imaging device 30. In the embodiment, the first infrared camera 31 and the second infrared camera 32 of the infrared imaging device 30 constitute a stereo camera. The infrared distance calculation unit 65 calculates the infrared distance Db from the infrared imaging device 30 to the second target by performing stereo processing on the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32.

[0049] FIG. 5 is a schematic diagram for explaining the visible light imaging device 20 and the infrared imaging device 30 according to the embodiment. As shown in FIG. 5(A), the visible light imaging device 20 images a predetermined range around the working machine 1. As shown in FIG. 5(B), the infrared imaging device 30 images a predetermined range around the working machine 1. At least a part of the imaging range of the visible light imaging device 20 coincides with at least a part of the imaging range of the infrared imaging device 30.

[0050] FIG. 5 shows a situation where an event in which the visible light image Ga captured by the visible light imaging device 20 becomes unclear does not occur. In FIG. 5, the first object captured by the visible light imaging device 20 and the second object captured by the infrared imaging device 30 are the same imaging object. The visible light distance calculation unit 64 calculates the visible light distance Da from the visible light imaging device 20 to the first object by performing stereo processing on the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. The infrared distance calculation unit 65 calculates the infrared distance Db from the infrared imaging device 30 to the second object by performing stereo processing on the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32. In FIG. 5, the visible light distance Da and the infrared distance Db are substantially equal.

[0051] FIG. 6 is a schematic diagram for explaining the visible light image Ga and the infrared image Gb according to the embodiment. FIG. 6 shows the visible light image Ga and the infrared image Gb captured when an event in which the visible light image Ga captured by the visible light imaging device 20 becomes unclear does not occur. In FIG. 6, the first object captured by the visible light imaging device 20 and the second object captured by the infrared imaging device 30 are the same imaging object. The first object appearing in the visible light image Ga and the second object appearing in the infrared image Gb are the same imaging object.

[0052] As shown in FIG. 6(A), a plurality of first section regions Pa are defined in the visible light image Ga. The first section regions Pa are defined in a matrix form in the visible light image Ga. In the embodiment, the first section region Pa includes the pixels (pixel regions) of the visible light image Ga. The pixels of the visible light image Ga correspond to the pixels of the image sensor of the visible light imaging device 20.

[0053] As shown in FIG. 6(B), a plurality of second partition regions Pb are defined in the infrared image Gb. A plurality of second partition regions Pb are defined in a matrix pattern in the infrared image Gb. In the embodiment, the second partition region Pb includes pixels (pixel regions) of the infrared image Gb. The pixels of the infrared image Gb correspond to the pixels of the image sensor of the infrared imaging device 30.

[0054] As shown in FIGS. 6(A) and 6(B), a reference point Po of the image region is set in each of the visible light image Ga and the infrared image Gb. The positions indicated by the reference points Po of both images coincide, and the first partition region Pa obtained by dividing the visible light image Ga and the second partition region Pb obtained by dividing the infrared image Gb are calibrated so that their sizes and positions (the ranges indicated by the images) coincide with each other with reference to the reference point Po.

[0055] In the embodiment, if the first partition region Pa and the second partition region Pb obtained by dividing the image coincide with each other, the structure of the image sensor of the visible light imaging device 20 and the structure of the image sensor of the infrared imaging device 30 may be different. For example, the size of the pixels of the image sensor of the visible light imaging device 20 and the size of the pixels of the image sensor of the infrared imaging device 30, the number of pixels (number of rows and columns) of the image sensor of the visible light imaging device 20 and the number of pixels (number of rows and columns) of the image sensor of the infrared imaging device 30, and the interval between adjacent pixels in the image sensor of the visible light imaging device 20 and the interval between adjacent pixels in the image sensor of the infrared imaging device 30 may be different. Also, the structure of the optical system of the visible light imaging device 20 and the structure of the image sensor of the infrared imaging device 30 may be different. For example, the focal length of the optical system of the visible light imaging device 20 and the focal length of the optical system of the infrared imaging device 30, and the field of view range of the optical system of the visible light imaging device 20 and the field of view range of the optical system of the infrared imaging device 30 may be different.

[0056] Therefore, the plurality of first partition regions Pa (pixels) defined in the visible light image Ga and the plurality of second partition regions Pb (pixels) defined in the infrared image Gb correspond to each other one-to-one, and the regions indicated by the pixels coincide with each other in position.

[0057] Therefore, the size of the first partition area Pa and the size of the second partition area Pb are equal. The number of the first partition areas Pa and the number of the second partition areas Pb are equal.

[0058] The visible light distance calculation unit 64 calculates a visible light distance Da for each of a plurality of first partition areas Pa defined in the visible light image Ga. The visible light distance calculation unit 64 calculates, for each of the plurality of first partition areas Pa of the visible light image Ga, the visible light distance Da from the visible light imaging device 20 to a first object existing in the imaging range of the visible light imaging device 20.

[0059] The infrared distance calculation unit 65 calculates an infrared distance Db for each of a plurality of second partition areas Pb defined in the infrared image Gb. The infrared distance calculation unit 65 calculates, for each of the plurality of second partition areas Pb of the infrared image Gb, the infrared distance Db from the infrared imaging device 30 to a second object existing in the imaging range of the infrared imaging device 30.

[0060] The determination unit 66 determines, for each of the mutually corresponding first partition area Pa and second partition area Pb, whether or not the difference between the visible light distance Da and the infrared distance Db is equal to or greater than a predetermined distance threshold.

[0061] FIG. 7 is a schematic diagram for explaining the visible light imaging device 20 and the infrared imaging device 30 according to the embodiment. FIG. 7 shows a situation in which an event occurs in which the visible light image Ga captured by the visible light imaging device 20 becomes unclear.

[0062] In the operation of the work machine 1, an event may occur in which the visible light image Ga becomes unclear. As shown in FIG. 7, an example of an event in which the visible light image Ga becomes unclear is the generation of dust due to the operation of the work machine 1. As shown in FIG. 7(A), at least a part of the dust may be generated in the space between the visible light imaging device 20 and the imaging target. As shown in FIG. 7(B), at least a part of the dust may be generated in the space between the infrared imaging device 30 and the imaging target.

[0063] Visible light cannot penetrate dust, and the visible light imaging device 20 cannot image an imaging target blocked by dust. The first target imaged by the visible light imaging device 20 includes dust and a part of the imaging target.

[0064] Infrared light can penetrate dust, and the infrared imaging device 30 can image an imaging target blocked by dust. The second target imaged by the infrared imaging device 30 hardly includes dust and includes the imaging target.

[0065] As described above, the visible light distance calculation unit 64 calculates the visible light distance Da from the visible light imaging device 20 to the first target by performing stereo processing on the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. The visible light distance Da calculated by the visible light distance calculation unit 64 includes the visible light distance Da from the visible light imaging device 20 to the dust and the visible light distance Da from the visible light imaging device 20 to the imaging target. The visible light distance Da from the visible light imaging device 20 to the dust is shorter than the visible light distance Da from the visible light imaging device 20 to the imaging target.

[0066] As described above, the infrared light distance calculation unit 65 calculates the infrared light distance Db from the infrared imaging device 30 to the second target by performing stereo processing on the infrared light image Gb captured by the first infrared light camera 31 and the infrared light image Gb captured by the second infrared light camera 32. The infrared light distance Db calculated by the visible light distance calculation unit 64 does not include the infrared light distance Db from the infrared imaging device 30 to the dust and includes the infrared light distance Db from the infrared imaging device 30 to the imaging target. The visible light distance Da from the visible light imaging device 20 to the dust is shorter than the infrared light distance Db from the infrared imaging device 30 to the imaging target.

[0067] As described above, when dust exists in the space between the visible light imaging device 20 and the infrared imaging device 30 and the imaging target, there may be a difference between the visible light distance Da and the infrared light distance Db. The determination unit 66 can determine the presence or absence of dust by determining whether the difference between the visible light distance Da and the infrared light distance Db is equal to or greater than the distance threshold. Further, the determination unit 66 can specify the position of the dust by determining whether the difference between the visible light distance Da and the infrared light distance Db is equal to or greater than the distance threshold for each of the mutually corresponding first partition region Pa and second partition region Pb.

[0068] FIG. 8 is a schematic diagram for explaining the visible light image Ga and the infrared image Gb according to the embodiment. FIG. 8 shows the visible light image Ga and the infrared image Gb captured when an event occurs in which the visible light image Ga captured by the visible light imaging device 20 becomes unclear. FIG. 8(A) shows the visible light image Ga captured in a situation where dust is generated as described with reference to FIG. 7(A). FIG. 8(B) shows the infrared image Gb captured in a situation where dust is generated as described with reference to FIG. 7(B).

[0069] As shown in FIG. 8, due to dust, the first object captured by the visible light imaging device 20 and the second object captured by the infrared imaging device 30 may not match. The first object captured by the visible light imaging device 20 includes dust and a part of the imaging target. The second object captured by the infrared imaging device 30 substantially does not include dust and includes the imaging target.

[0070] As shown in FIG. 8(A), in the visible light image Ga, a part of the imaging target blocked by dust may not appear. The visible light image Ga includes dust and a part of the imaging target.

[0071] As shown in FIG. 8(B), in the infrared image Gb, dust does not appear and the imaging target appears. The infrared image Gb substantially does not include dust and includes the imaging target.

[0072] The synthesizing unit 67 generates a synthesized image Gd by synthesizing a second partition region Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than a distance threshold value and a visible light image Ga.

[0073] In the embodiment, the second partition region Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold value is a plurality of second partition regions Pb included in the cutout region Gc of FIG. 8(B). The second partition region Pb of the cutout region Gc corresponds to the first partition region Pa in which dust appears in the visible light image Ga.

[0074] The second partition region Pb of the cutout region Gc is a second partition region Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold value. In the embodiment, the cutout region Gc indicates an aggregate of a plurality of second partition regions Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold value. The second partition region Pb outside the cutout region Gc is a second partition region Pb in which the difference between the visible light distance Da and the infrared distance Db is less than the distance threshold value.

[0075] Each of FIGS. 9 and 10 is a schematic diagram for explaining a method for generating the synthesized image Gd according to the embodiment.

[0076] As shown in FIG. 9(B), the synthesizing unit 67 cuts out the cutout region Gc from the infrared image Gb. That is, the synthesizing unit 67 cuts out a plurality of second partition regions Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold value from the infrared image Gb.

[0077] As shown in FIG. 9(A), the synthesizing unit 67 removes a plurality of first partition regions Pa corresponding to the cutout region Gc from the visible light image Ga. That is, the synthesizing unit 67 removes a plurality of first partition regions Pa in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold value from the visible light image Ga.

[0078] The first partition area Pa corresponding to the cutout area Gc is the first partition area Pa where the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold. The first partition area Pa not corresponding to the cutout area Gc is the first partition area Pa where the difference between the visible light distance Da and the infrared distance Db is less than the distance threshold.

[0079] As shown in FIG. 10, the synthesizing unit 67 synthesizes the cutout area Gc cut out from the infrared image Gb and the visible light image Ga from which a plurality of first partition areas Pa corresponding to the cutout area Gc have been removed to generate a synthesized image Gd. The synthesizing unit 67 synthesizes the cutout area Gc and the visible light image Ga so that the cutout area Gc including a plurality of second partition areas Pb where the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold overlaps the first partition area Pa of the visible light image Ga corresponding to the cutout area Gc. In the embodiment, the synthesizing unit 67 fits the cutout area Gc including a plurality of second partition areas Pb where the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold into the portion of the visible light image Ga from which the first partition area Pa has been removed.

[0080] Note that after fitting the cutout area Gc to the visible light image Ga (after overlapping), the synthesizing unit 67 may perform smoothing processing on the boundary between the visible light image Ga and the cutout area Gc.

[0081] As shown in FIG. 10, the synthesized image Gd includes a plurality of second partition areas Pb where the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold, and a plurality of first partition areas Pa where the difference between the visible light distance Da and the infrared distance Db is less than the distance threshold. That is, the synthesized image Gd is composed of a part of the visible light image Ga and a part of the infrared image Gb.

[0082] Based on the infrared distance Db calculated by the infrared distance calculation unit 65, the reference line generation unit 68 generates a reference line image Ge indicating the distance from the working machine 1.

[0083] The display output unit 69 outputs the composite image Gd generated by the composite unit 67 to the display device 50 so that the composite image Gd is displayed on the display device 50. Further, the display output unit 69 outputs the reference line image Ge generated by the reference line generation unit 68 to the display device 50 so that the reference line image Ge is displayed on the display device 50.

[0084] FIG. 11 is a schematic diagram showing the composite image Gd and the reference line image Ge displayed on the display device 50 according to the embodiment.

[0085] When the composite image Gd is displayed on the display device 50, even if dust is generated, the operator of the work machine 1 can check the display device 50 and recognize the situation around the work machine 1. Further, when the reference line image Ge is displayed on the display device 50, the operator of the work machine 1 can check the display device 50 and recognize the distance from the work machine 1 to the imaging target (construction target).

[0086] [Display method] FIG. 12 is a flowchart showing the display method according to the embodiment.

[0087] The visible light imaging device 20 images the periphery of the work machine 1. The infrared imaging device 30 images the periphery of the work machine 1. The image output unit 304 transmits visible light image data indicating the visible light image Ga imaged by the visible light imaging device 20 to the control device 60 via the communication device 7 and the communication system 400. The image output unit 304 transmits infrared image data indicating the infrared image Gb imaged by the infrared imaging device 30 to the control device 60 via the communication device 7 and the communication system 400.

[0088] The visible light image acquisition unit 62 acquires the visible light image Ga transmitted from the image output unit 304 (step S1).

[0089] The infrared image acquisition unit 63 acquires the infrared image Gb transmitted from the image output unit 304 (step S2).

[0090] The visible light distance calculation unit 64 calculates a visible light distance Da indicating the distance from the visible light imaging device 20 to the first target for each of a plurality of first section regions Pa defined in the visible light image Ga by performing stereo processing on the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22 (step S3).

[0091] The infrared distance calculation unit 65 calculates an infrared distance Db indicating the distance from the infrared imaging device 30 to the second target for each of a plurality of second section regions Pb defined in the infrared image Gb so as to correspond to the first section region Pa by performing stereo processing on the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32 (step S4).

[0092] The determination unit 66 determines whether or not the difference between the visible light distance Da and the infrared distance Db is equal to or greater than a predetermined distance threshold for each of the mutually corresponding first section region Pa and second section region Pb. That is, the determination unit 66 determines whether or not the difference between the visible light distance Da and the infrared distance Db is large for each of the mutually corresponding first section region Pa and second section region Pb. Further, the determination unit 66 determines whether or not there exists a second section region Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold (step S5).

[0093] In step S5, when it is determined that there exists a second section region Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold, that is, when it is determined that the difference between the visible light distance Da and the infrared distance Db is large (step S5: Yes), the determination unit 66 determines whether or not a cutout region Gc indicating an aggregate of a plurality of second section regions Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold is equal to or greater than a predetermined region threshold. That is, the determination unit 66 determines whether or not the size (ratio) of the cutout region Gc occupying the infrared image Gb is large (step S6).

[0094] In step S6, when it is determined that the cutout area Gc is equal to or greater than the area threshold (step S6: Yes), that is, when it is determined that the size of the cutout area Gc occupying the infrared image Gb is large, the synthesizing unit 67 synthesizes the cutout area Gc and the visible light image Ga to generate a synthesized image Gd (step S7).

[0095] Further, the reference line generation unit 68 generates a reference line image Ge indicating the distance from the working machine 1 based on the infrared distance Db calculated by the infrared distance calculation unit 65.

[0096] The display output unit 69 outputs the synthesized image Gd to the display device 50 so that the synthesized image Gd generated by the synthesizing unit 67 is displayed on the display device 50. Further, the display output unit 69 outputs the reference line image Ge to the display device 50 so that the reference line image Ge generated by the reference line generation unit 68 is displayed on the display device 50. Thereby, as shown in FIG. 11, the synthesized image Gd and the reference line image Ge are displayed on the display device 50 (step S8).

[0097] In step S5, when it is determined that there is no second section area Pb in which the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold (step S5: No), the display output unit 69 outputs the visible light image Ga to the display device 50 so that the visible light image Ga is displayed on the display device 50. Further, the display output unit 69 outputs the reference line image Ge to the display device 50 so that the reference line image Ge generated by the reference line generation unit 68 is displayed on the display device 50. Thereby, the visible light image Ga and the reference line image Ge are displayed on the display device 50 (step S9).

[0098] In step S6, when it is determined that the cutout area Gc is less than the area threshold, that is, when it is determined that the size of the cutout area Gc occupying the infrared image Gb is small (step S6: Yes), the display output unit 69 outputs the visible light image Ga to the display device 50 so that the visible light image Ga is displayed on the display device 50. Further, the display output unit 69 outputs the reference line image Ge to the display device 50 so that the reference line image Ge generated by the reference line generation unit 68 is displayed on the display device 50. Thereby, the visible light image Ga and the reference line image Ge are displayed on the display device 50 (step S9).

[0099] That is, in the embodiment, even if there is a second section area Pb where the difference between the visible light distance Da and the infrared distance Db is equal to or greater than the distance threshold, when the number of such second section areas Pb is small, that is, when the cutout area Gc is small, the composite image Gd is not displayed on the display device 50, and the visible light image Ga is displayed on the display device 50. In other words, even if dust is generated, when the area of the imaging target blocked by the dust is small, the composite image Gd is not displayed on the display device 50, and the visible light image Ga is displayed on the display device 50.

[0100] Note that the process of step S6 may be omitted.

[0101] [Computer System] FIG. 13 is a block diagram showing a computer system 1000 according to an embodiment. The above-described control device 60 includes the computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including a non-volatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory), a storage 1003, and an interface 1004 including an input / output circuit. The functions of the above-described control device 60 are stored in the storage 1003 as a computer program. The processor 1001 reads the computer program from the storage 1003, expands it in the main memory 1002, and executes the above-described processing according to the program. Note that the computer program may be distributed to the computer system 1000 via a network.

[0102] The computer program or the computer system 1000 can execute: acquiring a visible light image Ga showing an image of a first object captured by a visible light imaging device 20 that captures a predetermined range around the working machine 1 according to the above-described embodiment; acquiring an infrared image Gb showing an image of a second object captured by an infrared imaging device 30 that captures a predetermined range around the working machine 1; calculating a visible light distance Da indicating the distance from the visible light imaging device 20 to the first object for each of a plurality of first section regions Pa defined in the visible light image Ga; calculating an infrared distance Db indicating the distance from the infrared imaging device 30 to the second object for each of a plurality of second section regions Pb defined to correspond to the first section region Pa in the infrared image Gb; determining whether the difference between the visible light distance Da and the infrared distance Db is greater than or equal to a distance threshold for each corresponding first section region Pa and second section region Pb; generating a composite image Gd by synthesizing the second section region Pb where the difference is greater than or equal to the distance threshold and the visible light image Ga; and outputting the composite image Gd so that the composite image Gd is displayed on the display device 50.

[0103] [Effect] As described above, in a situation where no event occurs in which the visible light image Ga becomes unclear, as described in step S9, the visible light image Ga is displayed on the display device 50. The visible light image Ga is a color image, and the resolution of the visible light image Ga is higher than that of the infrared image Gb. That is, the visibility of the visible light image Ga is superior to that of the infrared image Gb. Therefore, in a situation where no event occurs in which the visible light image Ga becomes unclear, the display system 10 displays the visible light image Ga on the display device 50. Thereby, the display system 10 can provide the operator of the work machine 1 with the situation around the work machine 1.

[0104] In a situation where an event occurs in which the visible light image Ga becomes unclear, as described in step S8, a composite image Gd of the visible light image Ga and the infrared image Gb is displayed on the display device 50. Only a partially unclear area in the visible light image Ga is replaced by the infrared image Gb. As described above, the visibility of the visible light image Ga is excellent. Since only a partially unclear area of the visible light image Ga is replaced by the infrared image Gb instead of all areas of the visible light image Ga being replaced by the infrared image Gb, the visibility of the composite image Gd is maintained in a good state.

[0105] A partial area of the visible light image Ga to be replaced by the infrared image Gb is determined based on the difference between the visible light distance Da and the infrared distance Db. When an area where the difference between the visible light distance Da and the infrared distance Db is large exists in the visible light image Ga and the infrared image Gb, the visible light image Ga and the infrared image Gb are combined. Thereby, only a partially unclear area of the visible light image Ga is appropriately replaced by the infrared image Gb. The display system 10 can provide the operator of the work machine 1 with the situation around the work machine 1.

[0106] [Other Embodiments] In the above-described embodiment, it is assumed that the event that the image captured by the visible light imaging device 20 becomes unclear is the generation of dust. Examples of the event that the image captured by the visible light imaging device 20 becomes unclear include, in addition to the generation of dust, the generation of fog, insufficient visible light due to the work of the work machine being carried out at night, and the imaging target being in a backlight state.

[0107] When fog particles (water droplets) exist in the space between the visible light imaging device 20 and the infrared imaging device 30 and the imaging target, it is difficult for visible light to pass through the fog particles, but infrared light can pass through the fog particles. Therefore, the display system 10 can generate the composite image Gd according to the above-described embodiment. Even if an event occurs in which at least a part of the visible light image Ga captured by the visible light imaging device 20 becomes unclear, the display system 10 can provide the operator of the work machine 1 with the situation around the work machine 1 by displaying the composite image Gd on the display device 50.

[0108] In the case of insufficient visible light, the luminance of the pixels in the image sensor of the visible light imaging device 20 becomes too low. That is, so-called blackout occurs. In that case, it becomes difficult for the visible light distance calculation unit 64 to perform stereo processing on the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. That is, in the case of insufficient visible light, it becomes difficult for the visible light distance calculation unit 64 to calculate the visible light distance Da. On the other hand, the infrared distance calculation unit 65 can calculate the infrared distance Db by performing stereo processing on the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32. Even in the case of insufficient visible light, since a difference occurs between the visible light distance Da and the infrared distance Db, the display system 10 can generate the composite image Gd according to the above-described embodiment.

[0109] When the imaging target is in a backlight state, the luminance of the pixels in the image sensor of the visible light imaging device 20 becomes excessive. That is, so-called blooming occurs. In that case, it becomes difficult for the visible light distance calculation unit 64 to perform stereo processing on the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. That is, when the imaging target is in a backlight state, it becomes difficult for the visible light distance calculation unit 64 to calculate the visible light distance Da. On the other hand, the infrared distance calculation unit 65 can calculate the infrared distance Db by performing stereo processing on the infrared image Gb captured by the first infrared camera 31 and the infrared image Gb captured by the second infrared camera 32. Even when the imaging target is in a backlight state, since there is a difference between the visible light distance Da and the infrared distance Db, the display system 10 can generate the composite image Gd according to the above-described embodiment.

[0110] In the above-described embodiment, the visible light distance calculation unit 64 calculates the visible light distance Da by performing stereo processing on the visible light image Ga captured by the first visible light camera 21 and the visible light image Ga captured by the second visible light camera 22. A distance detection device such as a laser sensor (LIDAR: Light Detection and Ranging) may be provided on the work machine 1. The laser sensor detects the distance using laser light in the wavelength range of visible light. The visible light distance calculation unit 64 can calculate the visible light distance Da from the visible light imaging device 20 to the first target for each of a plurality of first section regions Pa defined in the visible light image Ga based on the detection data of the distance detection device. When the distance detection device is provided, one of the first visible light camera 21 and the second visible light camera 22 of the visible light imaging device 20 may be omitted.

[0111] In the above-described embodiment, one first partition region Pa may be one pixel of the visible light image Ga or a combination of a plurality of pixels. One second partition region Pb may be one pixel of the infrared image Gb or a combination of a plurality of pixels. Further, the first partition region Pa may be defined without depending on the pixels of the visible light image Ga. The second partition region Pb may be defined without depending on the pixels of the infrared image Gb. It is sufficient that a plurality of first partition regions Pa and a plurality of second partition regions Pb correspond one-to-one.

[0112] In the above-described embodiment, the display system 10 is applied to the remote operation system 100. The display device 50 does not have to be arranged in the remote operation room 200. The display device 50 may be arranged in the operation room (cab) of the work machine 1. Further, some functions of the control device 60 described in the above-described embodiment may be arranged in the work machine 1. An operator who has boarded the operation room of the work machine 1 can operate the boarding operation device arranged in the operation room of the work machine 1 while checking the display device 50 arranged in the operation room of the work machine 1. Also in this case, the display system 10 can provide the operator of the work machine 1 with the situation around the work machine 1 even if an event occurs in which the visible light image Ga captured by the visible light imaging device 20 becomes unclear.

[0113] In the above-described embodiment, the work machine 1 is a hydraulic excavator. The work machine 1 may be a bulldozer, a wheel loader, or a dump truck.

Description of Reference Numerals

[0114] 1... Construction machine, 2... Traveling body, 3... Slewing body, 4... Working machine, 4A... Boom, 4B... Arm, 4C... Bucket, 5... Hydraulic cylinder, 5A... Boom cylinder, 5B... Arm cylinder, 5C... Bucket cylinder, 6... Communication device, 7... Communication device, 10... Display system, 20... Visible light imaging device, 21... First visible light camera, 22... Second visible light camera, 30... Infrared imaging device, 31... First infrared camera, 32... Second infrared camera, 40... Remote control device, 45... Operator's seat, 50... Display device, 60... Control device, 61... Operation signal output section, 62... Visible light image acquisition section, 63... Infrared image acquisition section, 64... Visible light distance calculation section, 65... Infrared distance calculation section, 66... Judgment section, 67... Composition section, 68... Reference line generation section, 69... Display output section, 100... Remote control system, 200... Remote control cabin, 300... Control device, 301... Travel control section, 302... Slewing control section, 303... Working machine control section, 304... Image output section, 400... Communication system, Da... Visible light distance, Db... Infrared distance, Ga... Visible light image, Gb... Infrared image, Gc... Cut-out area, Gd... Composite image, Ge... Reference line image, Pa... First section area, Pb... Second section area, RX... Slewing axis.

Claims

1. a visible light image acquisition unit that acquires a visible light image indicating an image of a first object captured by a visible light imaging device provided in the work machine; an infrared image acquisition unit that acquires an infrared image indicating an image of a second object captured by an infrared imaging device provided on the work machine; a visible light distance calculation unit that calculates a visible light distance indicating a distance from the visible light imaging device to the first object for each of a plurality of first divided regions defined in the visible light image; an infrared distance calculation unit that calculates an infrared distance indicating a distance from the infrared imaging device to the second object for each of a plurality of second divided areas defined in the infrared image so as to correspond to the first divided areas; a determination unit that determines whether or not a difference between the visible light distance and the infrared distance is equal to or greater than a distance threshold for each of the corresponding first divided area and the corresponding second divided area; a synthesis unit that synthesizes the second divided area, in which the difference is equal to or greater than a distance threshold, with the visible light image to generate a synthetic image; A display output unit that outputs the composite image so that the composite image is displayed on a display device. Display system.

2. the combining unit combines the second divided region and the visible light image such that the second divided region in which the difference is equal to or greater than a distance threshold is superimposed on the first divided region of the visible light image corresponding to the second divided region. The display system of claim 1 .

3. the composite image includes the second divided region in which the difference is equal to or greater than a distance threshold and the first divided region in which the difference is less than the distance threshold, A display system according to claim 1 or 2.

4. the visible light imaging device includes a first visible light camera and a second visible light camera, the visible light distance calculation unit calculates the visible light distance by stereo processing a visible light image captured by the first visible light camera and a visible light image captured by the second visible light camera. A display system according to any one of claims 1 to 3.

5. The infrared imaging device includes a first infrared camera and a second infrared camera, the infrared distance calculation unit calculates the infrared distance by stereo processing an infrared image captured by the first infrared camera and an infrared image captured by the second infrared camera. A display system according to any one of claims 1 to 4.

6. a reference line generating unit that generates a reference line image indicating a distance from the work machine based on the infrared distance, The display output unit outputs the reference line image so that the reference line image is displayed on the display device. A display system according to any one of claims 1 to 5.

7. acquiring a visible light image representing an image of a first object captured by a visible light imaging device provided on the work machine; acquiring an infrared image representing an image of a second object captured by an infrared imaging device provided on the work machine; calculating a visible light distance indicating a distance from the visible light imaging device to the first object for each of a plurality of first divided regions defined in the visible light image; Calculating an infrared distance indicating a distance from the infrared imaging device to the second object for each of a plurality of second divided areas defined in the infrared image so as to correspond to the first divided areas; determining whether or not a difference between the visible light distance and the infrared distance is equal to or greater than a distance threshold for each of the corresponding first and second divided regions; synthesizing the second divided area, in which the difference is equal to or greater than a distance threshold, with the visible light image to generate a synthetic image; and outputting the composite image so that the composite image is displayed on a display device. Display method.

Images