Display system and display method

Abstract

To provide an operator of a work machine with surrounding conditions of the work machine even when a synthesized image is unclear.SOLUTION: A display system comprises: a first deforming unit for generating a plurality of first deformed images from a first image obtained by means of imaging performed by a first imaging device; a first synthesis unit for generating a plurality of first synthesized images by synthesizing a second image obtained by means of imaging performed by a second imaging device with each of the plurality of first deformed images; a selecting unit for selecting a predetermined first synthesized image from among the plurality of first synthesized images; and a display control unit for causing a display device to display a display image generated on the basis of the selected first synthesized image.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present disclosure relates to a display system and a display method.

Background Art

[0002] An image synthesis system as disclosed in Patent Document 1 is known. An image synthesis method as disclosed in Non-Patent Document 1 is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the field of engineering machinery, a technique is known in which the area surrounding the machinery is imaged using a visible light imaging device and an infrared imaging device, and the combined visible light and infrared images are provided to the operator of the machinery. By checking the combined image of the visible light and infrared images, the operator can confirm the surrounding conditions of the machinery even at night or in backlit conditions. On the other hand, if the visible light imaging device and the infrared imaging device are installed at a physical distance from each other, parallax may cause image shift in the combined image of the visible light and infrared images, potentially resulting in an unclear image.

[0006] This disclosure aims to provide the operator of a work machine with information about the surrounding environment of the work machine, even if the composite image becomes unclear. [Means for solving the problem]

[0007] A display system is provided, comprising: a first deformation unit that generates a plurality of first deformation images from a first image obtained by imaging with a first imaging device; a first synthesis unit that generates a plurality of first composite images by synthesizing a second image obtained by imaging with a second imaging device and each of the plurality of first deformation images; a selection unit that selects a predetermined first composite image from the plurality of first composite images; and a display control unit that causes a display image generated based on the selected first composite image to be displayed on a display device. [Effects of the Invention]

[0008] According to this disclosure, even if an event occurs that causes the composite image to become unclear, the operator of the work machine can be provided with information about the surrounding environment of the work machine. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram showing a remote control system for a work machine according to an embodiment. [Figure 2] Figure 2 is a perspective view showing a work machine according to an embodiment. [Figure 3] Figure 3 is a perspective view showing a visible light imaging device and a far-infrared imaging device according to an embodiment. [Figure 4] FIG. 4 is a diagram schematically showing a visible light imaging device according to an embodiment. [Figure 5] FIG. 5 is a functional block diagram showing a remote operation system of a working machine according to an embodiment. [Figure 6] FIG. 6 is a diagram schematically showing an example of a state in which a visible light imaging device according to an embodiment is imaging an imaging target. [Figure 7] FIG. 7 is a diagram schematically showing an example of a state in which an infrared imaging device according to an embodiment is imaging an imaging target. [Figure 8] FIG. 8 is a diagram schematically showing an outline of processing of an image processing unit according to an embodiment. [Figure 9] FIG. 9 is a diagram for explaining the influence of the parallax between the visible light imaging device and the infrared imaging device on the processing of the image processing unit. [Figure 10] FIG. 10 is a functional block diagram showing an image processing unit according to an embodiment. [Figure 11] FIG. 11 is a functional block diagram showing an alignment processing unit according to an embodiment. [Figure 12] FIG. 12 is a diagram schematically showing an outline of processing of the alignment processing unit according to an embodiment. [Figure 13] FIG. 13 is a flowchart showing a display method according to an embodiment. [Figure 14] FIG. 14 is a block diagram showing a computer system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, but the present disclosure is not limited to the embodiments. The components of the embodiments described below can be combined as appropriate. Also, some components may not be used.

[0011] [Remote operation system] FIG. 1 is a diagram schematically showing a remote operation system 2 of a working machine 1 according to an embodiment. The remote operation system 2 remotely operates the working machine 1 existing at the work site. At least a part of the remote operation system 2 is arranged in a remote operation room 3 at the remote operation location. The remote operation system 2 includes a remote operation device 4, a display device 5, and a control device 6.

[0012] The remote operation device 4 is arranged in the remote operation room 3 outside the working machine 1. The remote operation device 4 is operated by an operator in the remote operation room 3. The operator can operate the remote operation device 4 while sitting on the operator's seat 7.

[0013] The display device 5 is arranged in the remote operation room 3 outside the working machine 1. The display device 5 displays an image of the work site. The image of the work site includes an image of a predetermined range around the working machine 1. The image of the predetermined range around the working machine 1 includes at least an image of the work target of the working machine 1. The work target of the working machine 1 includes the construction target of the working machine 1.

[0014] The display device 5 includes a panel display such as a liquid crystal display (LCD) or an organic EL display (OELD). In the embodiment, the display device 5 includes a plurality of flat panel displays arranged adjacent to each other. Note that the display device 5 may be constituted by one flat panel display. The display device 5 may be constituted by a curved display or a screen.

[0015] The operator operates the remote operation device 4 while checking the image of the work site displayed on the display device 5. The working machine 1 is remotely operated by the remote operation device 4.

[0016] The control device 6 is arranged in the remote operation room 3 outside the working machine 1. The control device 6 includes a computer system.

[0017] The work machine 1 is equipped with a control device 8. The control device 8 includes a computer system.

[0018] Control device 6 and control device 8 communicate via communication system 9. Examples of communication system 9 include the internet, local area network (LAN), mobile phone network, and satellite network.

[0019] [Working machinery] Figure 2 is a perspective view showing a work machine 1 according to an embodiment. In this embodiment, the work machine 1 is a hydraulic excavator. The work machine 1 operates at the work site.

[0020] As shown in Figure 2, the work machine 1 comprises a traveling body 10, a rotating body 11 supported by the traveling body 10, a work machine 12 supported by the rotating body 11, a hydraulic cylinder 13 that drives the work machine 12, a visible light imaging device 14, and an infrared imaging device 15.

[0021] The traveling body 10 is capable of traveling while supporting the slewing body 11. The slewing body 11 is capable of rotatable around the slewing axis RX while being supported by the traveling body 10. The work equipment 12 includes a boom 12A rotatably connected to the slewing body 11, an arm 12B rotatably connected to the boom 12A, and a bucket 12C rotatably connected to the arm 12B. The hydraulic cylinder 13 includes a boom cylinder 13A that drives the boom 12A, an arm cylinder 13B that drives the arm 12B, and a bucket cylinder 13C that drives the bucket 12C.

[0022] In this embodiment, the direction parallel to the slewing axis RX is appropriately referred to as the up-down direction, the direction parallel to the rotation axis of the work implement 12 is appropriately referred to as the left-right direction, and the direction perpendicular to both the slewing axis RX and the rotation axis of the work implement 12 is appropriately referred to as the front-rear direction. With respect to the slewing axis RX, the direction in which the work implement 12 is located is forward, and the opposite direction of forward is rear. With respect to the slewing axis RX, one of the left-right directions is to the right, and the opposite direction of right is to the left. The direction away from the ground contact surface of the traveling body 10 is upward, and the opposite direction of upward is downward.

[0023] [Visible light imaging device and far-infrared imaging device] Figure 3 is a perspective view showing a visible light imaging device 14 and an infrared imaging device 15 according to an embodiment. The visible light imaging device 14 and the infrared imaging device 15 are each positioned on the work machine 1. In this embodiment, the visible light imaging device 14 and the infrared imaging device 15 are each positioned on the upper front part of the rotating body 11. The visible light imaging device 14 and the infrared imaging device 15 simultaneously image the work site in front of the rotating body 11.

[0024] The visible light imaging device 14 and the infrared imaging device 15 are arranged adjacent to each other in the work machine 1. In this embodiment, the visible light imaging device 14 is positioned to the right of the infrared imaging device 15. Alternatively, the visible light imaging device 14 may be positioned to the left of the infrared imaging device 15. Both the visible light imaging device 14 and the infrared imaging device 15 are fixed to the rotating body 11. The relative positions of the visible light imaging device 14 and the infrared imaging device 15 are constant.

[0025] The visible light imaging device 14 includes a visible light camera that acquires images in the visible light wavelength range. The visible light wavelength range is, for example, 360 nm to 830 nm.

[0026] The infrared imaging device 15 includes an infrared camera capable of acquiring images within the infrared spectral range. The infrared spectral range is between 780 nm and 100 μm. In an embodiment, the infrared imaging device 15 acquires images within the far-infrared spectral range. The spectral range of the infrared imaging device 15 is, for example, between 7.5 μm and 14 μm.

[0027] The visible light imaging device 14 and the infrared imaging device 15 each capture images of objects present around the work machine 1. The objects to be captured are objects. Examples of objects to be captured by the visible light imaging device 14 and the infrared imaging device 15 include the work area of ​​work machine 1, the excavation area of ​​work machine 12, structures present at the work site, at least a part of work machine 1, work machines other than work machine 1, and people (workers) working at the work site.

[0028] Figure 4 is a schematic diagram showing a visible light imaging device 14 according to an embodiment. The visible light imaging device 14 includes an optical system 14A and an image sensor 14B that receives light that has passed through the optical system 14A. Examples of the image sensor 14B include a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The optical axis AX1 of the optical system 14A extends substantially in the front-to-back direction. The imaging surface 14C of the image sensor 14B is substantially perpendicular to the optical axis AX1.

[0029] Similar to the visible light imaging device 14, the infrared imaging device 15 includes an optical system 15A and an image sensor 15B. The optical axis AX2 of the optical system 15A extends substantially in the front-to-back direction. The imaging surface 15C of the image sensor 15B is substantially perpendicular to the optical axis AX2.

[0030] The visible light imaging device 14 and the infrared imaging device 15 are fixed to the work machine 1 such that the optical axes AX1 and AX2 are approximately parallel.

[0031] The visible light imaging device 14 images the object to be imaged, which is positioned within the imaging range of the visible light imaging device 14. The infrared imaging device 15 images the object to be imaged, which is positioned within the imaging range of the infrared imaging device 15. The imaging range of the visible light imaging device 14 and at least a portion of the imaging range of the infrared imaging device 15 coincide. The imaging range of the visible light imaging device 14 includes the field of view of the optical system 14A of the visible light imaging device 14. The imaging range of the infrared imaging device 15 includes the field of view of the optical system 15A of the infrared imaging device 15. In this embodiment, the imaging range of the visible light imaging device 14 and the imaging range of the infrared imaging device 15 coincide. However, the imaging range of the visible light imaging device 14 and a portion of the imaging range of the infrared imaging device 15 may coincide.

[0032] In the following description, images captured by the visible light imaging device 14 will be referred to as visible light images Ga, and images captured by the infrared imaging device 15 will be referred to as infrared images Gb.

[0033] In the following description, the direction substantially parallel to the optical axis AX1 of the visible light imaging device 14 and the optical axis AX2 of the infrared imaging device 15 will be appropriately referred to as the depth direction, and the direction intersecting the optical axes AX1 and AX2 will be appropriately referred to as the screen direction.

[0034] In this embodiment, the screen direction is substantially parallel to the imaging surface 14C of the visible light imaging device 14 and the imaging surface 15C of the infrared imaging device 15, respectively. The depth direction is equal to the front-to-back direction. The screen direction is equal to the left-to-right direction.

[0035] [Display System] Figure 5 is a functional block diagram showing the remote control system 2 of the work machine 1 according to this embodiment.

[0036] The remote control system 2 comprises a communication device 16 located at the remote control location, a control device 6 connected to the communication device 16, a remote control device 4 connected to the control device 6, and a display device 5 connected to the control device 6.

[0037] Furthermore, the remote control system 2 includes a communication device 17 located on the work machine 1, a control device 8 connected to the communication device 17, a visible light imaging device 14 connected to the control device 8, an infrared imaging device 15 connected to the control device 8, a traveling body 10 controlled by the control device 8, a rotating body 11 controlled by the control device 8, and a hydraulic cylinder 13 controlled by the control device 8.

[0038] The remote control system 2 has a display system 18 that displays images of the work site. The display system 18 includes a visible light imaging device 14, an infrared imaging device 15, a control device 6, and a display device 5.

[0039] The control device 8 includes a traveling body control unit 19, a rotating body control unit 20, a work machine control unit 21, and an image output unit 22.

[0040] The vehicle control unit 19 receives the operation signal from the remote control device 4 transmitted from the control device 6. Based on the operation signal from the remote control device 4, the vehicle control unit 19 outputs a control signal to control the operation of the vehicle 10.

[0041] The slewing body control unit 20 receives the operation signal from the remote control device 4 transmitted from the control device 6. Based on the operation signal from the remote control device 4, the slewing body control unit 20 outputs a control signal to control the movement of the slewing body 11.

[0042] The work equipment control unit 21 receives operation signals from the remote control device 4 transmitted from the control device 6. Based on the operation signals from the remote control device 4, the work equipment control unit 21 outputs control signals to control the operation of the work equipment 12. The control signals for controlling the work equipment 12 include control signals for controlling the hydraulic cylinder 13.

[0043] The image output unit 22 outputs visible light image data showing the visible light image Ga captured by the visible light imaging device 14. The image output unit 22 also outputs infrared image data showing the infrared image Gb captured by the infrared imaging device 15.

[0044] Communication device 17 communicates with communication device 16 via communication system 9. Communication device 17 receives operation signals from remote control device 4 transmitted from control device 6 via communication device 16 and outputs them to control device 8. Communication device 17 transmits visible light image data and infrared image data output from image output unit 22 to communication device 16. Communication device 17 includes encoders for compressing the visible light image data and infrared image data, respectively. The visible light image data and infrared image data are transmitted from communication device 17 to communication device 16 in a compressed state.

[0045] Communication device 16 communicates with communication device 17 via communication system 9. Communication device 16 transmits operation signals generated by the operation of remote control device 4 to communication device 17. Communication device 16 receives visible light image data and infrared image data transmitted from control device 8 via communication device 17 and outputs them to control device 6. Communication device 16 includes a decoder that restores the compressed visible light image data and infrared image data, respectively. The restored visible light image data and infrared image data are output from communication device 16 to control device 6.

[0046] The control device 6 includes an operation signal output unit 23, a visible light image acquisition unit 24, an infrared image acquisition unit 25, a deformation parameter acquisition unit 26, an image processing unit 27, and a display control unit 28.

[0047] The operation signal output unit 23 outputs operation signals for remotely controlling the work machine 1. When the remote control device 4 is operated by the operator, operation signals for remotely controlling the work machine 1 are generated. The operation signals include operation signals for remotely controlling the traveling body 10, operation signals for remotely controlling the rotating body 11, and operation signals for remotely controlling the work machine 12. The operation signal output unit 23 outputs the operation signals from the remote control device 4. The communication device 16 transmits the operation signals output from the operation signal output unit 23 to the communication device 17.

[0048] The visible light image acquisition unit 24 acquires a visible light image Ga showing the image of the target being captured by the visible light imaging device 14. The visible light image acquisition unit 24 acquires the visible light image Ga by acquiring the visible light image data restored by the communication device 16.

[0049] The infrared image acquisition unit 25 acquires an infrared image Gb showing the image of the target being captured by the infrared imaging device 15. The infrared imaging device 15 acquires the infrared image Gb by acquiring the infrared image data restored by the communication device 16.

[0050] The deformation parameter acquisition unit 26 acquires deformation parameters that represent the deformation of the image. The deformation parameters are predetermined according to the conditions of the work site. For example, the deformation parameters may be input to the control device 6 via an input device (not shown). The deformation parameter acquisition unit 26 may acquire the deformation parameters input from the input device. In this embodiment, the deformation parameters hold a deformation parameter set that includes a plurality of deformation parameters.

[0051] The image processing unit 27 performs image processing based on the visible light image Ga and the infrared image Gb to generate a predetermined display image. The image processing unit 27 updates the display image at predetermined time intervals.

[0052] The display control unit 28 displays the display image generated by the image processing unit 27 on the display device 5. The operator operates the remote control device 4 while checking the display image displayed on the display device 5.

[0053] [Overview of Image Processing] Figure 6 is a schematic diagram showing an example of the visible light imaging device 14 according to the embodiment in which imaging is being performed on an object. Figure 7 is a schematic diagram showing an example of the infrared imaging device 15 according to the embodiment in which imaging is being performed on an object.

[0054] During operation of the work machine 1, there is a possibility that the visible light image Ga captured by the visible light imaging device 14 may become unclear. An example of an event that causes the visible light image Ga to become unclear is the generation of dust caused by the operation of the work machine 1. As shown in Figures 6 and 7, at least a portion of the dust may be generated in the space between the visible light imaging device 14 and the infrared imaging device 15 and the object being imaged.

[0055] Visible light cannot penetrate dust. The visible light imaging device 14 cannot image objects that are obscured by dust. The visible light image Ga captured by the visible light imaging device 14 includes dust and a portion of the object being imaged.

[0056] Infrared light can penetrate dust. The infrared imaging device 15 can image objects that are obscured by dust. The infrared image Gb captured by the infrared imaging device 15 contains virtually no dust and includes the entire object being imaged.

[0057] Thus, if dust is present in the space between the visible light imaging device 14 and the infrared imaging device 15 and the object being imaged, the entire object may be captured in the infrared image Gb, but a portion of the object may not be captured in the visible light image Ga.

[0058] The image processing unit 27 performs a process to fill in the portion of the visible light image Ga in which the imaging target was not captured with the infrared image Gb.

[0059] Figure 8 is a schematic diagram showing the overview of the processing of the image processing unit 27 according to the embodiment. As shown in Figure 8, a part of the object being captured is not captured in the visible light image Ga, while the entire object being captured is captured in the infrared image Gb. The image processing unit 27 generates a gradient image Gc of the visible light image Ga from the visible light image Ga. The image processing unit 27 also generates a gradient image Gd of the infrared image Gb from the infrared image Gb. The gradient image Gc includes an image in which the edges of the object being captured are extracted from the visible light image Ga. The gradient image Gd includes an image in which the edges of the object being captured are extracted from the infrared image Gb. Since a part of the object being captured is not captured in the visible light image Ga, edges are extracted in the gradient image Gc for the parts in which the object is captured, but not for the parts in which the object is not captured. Since the entire object being captured is captured in the infrared image Gb, all the edges of the object being captured are extracted in the gradient image Gd. The image processing unit 27 combines the gradient image Gc and the gradient image Gd to generate a composite gradient image Ge. The image processing unit 27 combines color information with the composite gradient image Ge to generate a composite image Gf. As a result, a portion of the visible light image Ga in which the imaging target was not captured is filled in with the infrared image Gb, and the composite image Gf is generated. The composite image Gf is displayed on the display device 5 as a display image.

[0060] A common camera coordinate system is defined for the visible light imaging device 14 and the infrared imaging device 15. The image processing unit 27 combines the gradient image Gc and the gradient image Gd in the camera coordinate system to generate a composite gradient image Ge. Combining the gradient image Gc and the gradient image Gd includes superimposing the gradient image Gc and the gradient image Gd.

[0061] The example shown in Figure 8 illustrates the processing of the image processing unit 27 when it is assumed that there is no physical distance between the visible light imaging device 14 and the infrared imaging device 15, and that the optical axes AX1 and AX2 coincide. In reality, as explained with reference to Figure 3, the visible light imaging device 14 and the infrared imaging device 15 are positioned separately in the left-right direction.

[0062] [Effect of parallax] Figure 9 is a diagram illustrating the effect of the parallax between the visible light imaging device 14 and the infrared imaging device 15 on the processing of the image processing unit 27. In this embodiment, the visible light imaging device 14 and the infrared imaging device 15 are positioned far apart in the left-right direction. As a result, due to the effect of the parallax between the visible light imaging device 14 and the infrared imaging device 15 with respect to the imaging target, simply combining the gradient image Gc and the gradient image Gd may result in the occurrence of ghost edges in the combined image Gf, as shown in Figure 9. In other words, due to the effect of the parallax between the visible light imaging device 14 and the infrared imaging device 15 with respect to the imaging target, the gradient image Gc and the gradient image Gd may be combined in a misaligned state. When the combined image Gf with ghost edges is displayed as the display image on the display device 5, it becomes difficult for the operator to recognize the situation at the work site.

[0063] Therefore, the image processing unit 27 deforms the gradient image Gd using deformation parameters to generate a deformed image, and then synthesizes the deformed image with the visible light image Ga. This suppresses the generation of ghost edges in the synthesized image Gf.

[0064] [Image Processing Department] Figure 10 is a functional block diagram showing the image processing unit 27 according to the embodiment. As shown in Figure 10, the image processing unit 27 includes a luminance and color information separation unit 29, a positioning processing unit 30, an image deformation unit 31, an image synthesis unit 32, and a luminance and color information synthesis unit 33.

[0065] The image processing unit 27 receives a visible light image Ga acquired by the visible light image acquisition unit 24, an infrared image Gb acquired by the infrared image acquisition unit 25, and a set of deformation parameters acquired by the deformation parameter acquisition unit 26. The set of deformation parameters includes multiple deformation parameters that represent the deformation of the image.

[0066] The luminance and color information separation unit 29 receives a visible light image Ga as input. The luminance and color information separation unit 29 separates the visible light image Ga into a luminance image and color information (color component image). The luminance image is a grayscale image.

[0067] The alignment processing unit 30 receives the luminance image, the infrared image Gb, and the deformation parameter set as inputs. The alignment processing unit 30 outputs the corrected deformation parameters.

[0068] The image deformation unit 31 receives the correction deformation parameters and the infrared image Gb as input. The image deformation unit 31 deforms the infrared image Gb based on the correction deformation parameters.

[0069] The image synthesis unit 32 (second synthesis unit) receives the luminance image and the infrared image Gb transformed by the image deformation unit 31 as input. The image synthesis unit 32 synthesizes the luminance image, the transformed infrared image Gb, and the luminance image to generate a composite luminance image. The image synthesis unit 32 synthesizes the luminance image and the transformed infrared image Gb based on an existing image synthesis method. An example of an existing image synthesis method is the image synthesis method described in Non-Patent Document 1.

[0070] The luminance and color information synthesis unit 33 receives a synthesized luminance image and color information (color component image) as input. The luminance and color information synthesis unit 33 synthesizes the synthesized luminance image and color information to generate a synthesized image Gf. The luminance and color information synthesis unit 33 synthesizes the synthesized luminance image and color information based on an existing image synthesis method. An example of an existing image synthesis method is the image synthesis method described in Non-Patent Document 1.

[0071] The display control unit 28 causes the composite image Gf to be displayed on the display device 5 as the display image.

[0072] Note that the infrared image Gb may be any grayscale image. If the visible light image Ga is a grayscale image, the luminance color information separation unit 29 and the luminance color information synthesis unit 33 are omitted.

[0073] Figure 11 is a functional block diagram showing the alignment processing unit 30 according to the embodiment. Figure 12 is a schematic diagram showing an overview of the processing of the alignment processing unit 30 according to the embodiment.

[0074] As shown in Figure 11, the alignment processing unit 30 includes a gradient information extraction unit 34, a gradient information extraction unit 35, an image deformation unit 36, a gradient synthesis unit 37, and a gradient evaluation / parameter selection unit 38.

[0075] The alignment processing unit 30 receives a luminance image, an infrared image Gb, and a set of deformation parameters as input.

[0076] The gradient information extraction unit 34 receives a luminance image as input. That is, the input image to the gradient information extraction unit 34 is a luminance image. The gradient information extraction unit 34 generates a gradient image Gc of the luminance image. The gradient information extraction unit 34 extracts gradient information for each pixel of the input luminance image and generates a gradient image Gc of the luminance image.

[0077] The input image input to the gradient information extraction unit 34 will be represented as u(x,y). (x,y) represents the image coordinates. u(x,y) represents the brightness value at image coordinates (x,y). Furthermore, the gradient information of the image coordinate position (x,y) is represented as g(x,y). The gradient information g(x,y) is defined by the following equation (1).

[0078]

number

[0079] In equation (1), L(·) represents the Laplacian filter, and |·| represents the absolute value.

[0080] The gradient information g(x,y) may also be defined by equation (2) below.

[0081]

number

[0082] In equation (2), d x 2 (x,y) represents the difference in the x direction, and d y 2(x,y) represents the difference in the y-direction. For the difference, forward difference, backward difference, or central difference can be used.

[0083] The gradient information extraction unit 35 receives the infrared image Gb as input. That is, the input image to the gradient information extraction unit 35 is the infrared image Gb. The gradient information extraction unit 35 generates a gradient image Gd of the infrared image Gb. The gradient information extraction unit 35 extracts gradient information for each pixel of the input image, the infrared image Gb, to generate a gradient image Gd of the infrared image Gb. Similar to the gradient information of the luminance image, the gradient information of the infrared image Gb is defined by equation (1) or (2) above.

[0084] The gradient information extraction unit 34 generates a gradient image Gc of the luminance image using the gradient information g(x,y) of the luminance image extracted for each pixel. The gradient information extraction unit 35 generates a gradient image Gd of the infrared image Gb using the gradient information g(x,y) of the infrared image Gb extracted for each pixel. The gradient image Gc is an image in which the edges of the imaged object are extracted from the luminance image. The gradient image Gd is an image in which the edges of the imaged object are extracted from the infrared image Gb.

[0085] The image deformation unit 36 ​​receives the gradient image Gd of the infrared image Gb and a deformation parameter set as input. The image deformation unit 36 ​​generates multiple deformed gradient images from the gradient image Gd of the infrared image Gb obtained by imaging by the infrared imaging device 15. The image deformation unit 36 ​​deforms the gradient image Gd to generate a deformed gradient image.

[0086] In this embodiment, deforming the gradient image Gd includes shifting the gradient image Gd in a predetermined shift direction in the camera coordinate system. The image deformation unit 36 ​​shifts the gradient image Gd in a predetermined shift direction in the camera coordinate system to generate a plurality of deformed gradient images.

[0087] In the embodiment, the image deformation unit 36 ​​generates a plurality of deformed gradient images from the gradient image Gd of the infrared image Gb based on a deformation parameter set. That is, the image deformation unit 36 ​​deforms the gradient image Gd based on each of the plurality of deformation parameters to generate a plurality of deformed gradient images. The deformation parameter includes a shift amount in a predetermined shift direction. In the embodiment, the deformation parameter set includes a plurality of shift amounts in the screen direction, defined for each of the plurality of positions in the depth direction. A plurality of shift amounts are defined for each of the plurality of positions in the depth direction.

[0088] As shown in Figure 12, the image deformation unit 36 ​​is located at multiple positions in the depth direction (d1, d2, ... d N ) is assumed. The image deformation unit 36 ​​shifts the gradient image Gd at each of the multiple positions in the depth direction and each of the multiple positions in the screen direction to generate multiple deformed gradient images. Based on the deformation parameter set, the image deformation unit 36 ​​shifts the gradient image Gd in the left and right directions by different amounts. In the example shown in Figure 12, among the multiple deformation parameters included in the deformation parameter set, the first deformed gradient image Gd1 is generated based on the first deformation parameter, the second deformed gradient image Gd2 is generated based on the second deformation parameter, the third deformed gradient image Gd3 is generated based on the third deformation parameter, and the fourth deformed gradient image Gd4 is generated based on the fourth deformation parameter. Each of the first, second, third, and fourth deformed gradient images Gd1, Gd2, Gd3, and Gd4 is an image in which the amount of left and right shift from the gradient image Gd is different at a certain position in the depth direction.

[0089] Note that the deformation gradient images (Gd1, Gd2, Gd3, Gd4) are not limited to four. There may be two or three deformation gradient images, or any number of five or more.

[0090] The gradient merging unit 37 receives the gradient image Gc of the luminance image and multiple deformed gradient images Gd1, Gd2, Gd3, and Gd4 as input. The gradient merging unit 37 generates multiple composite gradient images by merging the gradient image Gc of the luminance image with each of the multiple deformed gradient images Gd1, Gd2, Gd3, and Gd4 in the camera coordinate system. As shown in Figure 12, the first composite gradient image Ge1 is generated by merging the gradient image Gc with the first deformed gradient image Gd1. The second composite gradient image Ge2 is generated by merging the gradient image Gc with the second deformed gradient image Gd2. The third composite gradient image Ge3 is generated by merging the gradient image Gc with the third deformed gradient image Gd3. The fourth composite gradient image Ge4 is generated by merging the gradient image Gc with the fourth deformed gradient image Gd4.

[0091] Let's consider combining two gradient images g1(x,y) and g2(x,y). Let the resulting gradient image be G(x,y). The combined gradient image is expressed by equation (3) below.

[0092]

number

[0093] The gradient evaluation and parameter selection unit 38 receives a set of deformation parameters and multiple composite gradient images (in the example shown in Figure 12, the first, second, third, and fourth composite gradient images Ge1, Ge2, Ge3, and Ge4). The gradient evaluation and parameter selection unit 38 evaluates the composite gradient image generated for each deformation parameter. Based on the evaluation index, the gradient evaluation and parameter selection unit 38 selects a predetermined composite gradient image from the multiple composite gradient images. The gradient evaluation and parameter selection unit 38 outputs the deformation parameters corresponding to the composite gradient image with the best evaluation index as modified deformation parameters.

[0094] The gradient evaluation / parameter selection unit 38 sets a region of interest in the composite gradient image to evaluate the gradient. The gradient evaluation / parameter selection unit 38 uses gradient energy, which represents the average gradient value of the region of interest, as an evaluation index for the gradient. A smaller gradient energy indicates a better evaluation, while a larger gradient energy indicates a worse evaluation. A smaller gradient energy means that edges occupy a smaller proportion of the composite gradient image. A larger gradient energy means that edges occupy a larger proportion of the composite gradient image. In the third composite gradient image Ge3 shown in Figure 12, the edges of gradient image Gc and the edges of the third gradient image Gd3 overlap. Therefore, the proportion of edges in the third composite gradient image Ge3 is small. In other words, there are few ghost edges in the third composite gradient image Ge. Consequently, the gradient energy of the third composite gradient image Ge3 is small, and the evaluation is good. On the other hand, for example, in the first composite gradient image Ge1, the edges of gradient image Gc and the edges of the first gradient image Gd1 are misaligned. Therefore, edges make up a large proportion of the first composite gradient image Ge1. In other words, there are many ghost edges in the first composite gradient image Ge1. Consequently, the gradient energy of the first composite gradient image Ge1 is large, and the evaluation is poor. Similarly, edges make up a large proportion of the second and fourth composite gradient images Ge2 and G24. The gradient energy of the second and fourth composite gradient images Ge2 and Ge4 is large, and the evaluation is poor.

[0095] In the example shown in Figure 12, the gradient evaluation and parameter selection unit 38 selects the third composite gradient image Ge3, which has the smallest gradient energy, from among the multiple composite gradient images Ge1, Ge2, Ge3, and Ge4. The gradient evaluation and parameter selection unit 38 outputs the deformation parameters used to generate the selected third composite gradient image Ge3, i.e., the deformation parameters used to generate the third deformed gradient image Gd3, as modified deformation parameters.

[0096] In the example shown in Figure 11, a gradient image Gd is generated from the infrared image Gb, and then a deformed gradient image is generated from the gradient image Gd. Alternatively, the infrared image Gb may be deformed to generate a deformed infrared image, and then gradient information may be extracted from the deformed infrared image to generate a gradient image.

[0097] [Display method] Figure 13 is a flowchart showing the display method according to the embodiment.

[0098] The deformation parameter acquisition unit 26 acquires a predetermined set of deformation parameters (step S1).

[0099] The visible light imaging device 14 and the infrared imaging device 15 each capture an image of the target object at the work site. The image output unit 22 transmits visible light image data, showing the visible light image Ga captured by the visible light imaging device 14, to the control device 6 via the communication device 17 and the communication system 9. The image output unit 22 also transmits infrared image data, showing the infrared image Gb captured by the infrared imaging device 15, to the control device 6 via the communication device 17 and the communication system 9.

[0100] The visible light image acquisition unit 24 acquires the visible light image Ga transmitted from the image output unit 22. The infrared image acquisition unit 25 acquires the infrared image Gb transmitted from the image output unit 22 (step S2).

[0101] The luminance and color information separation unit 29 separates the visible light image Ga into a luminance image and color information (step S3).

[0102] The gradient information extraction unit 34 generates a gradient image Gc of the luminance image. The gradient information extraction unit 35 generates a gradient image Gd of the infrared image Gb (step S4).

[0103] The image deformation unit 36 ​​(first deformation unit) generates multiple deformation gradient images (first deformation images) from the gradient image Gd (first image) of the infrared image Gb obtained by imaging with the infrared imaging device 15, based on the deformation parameter set. In the example shown in Figure 12, the image deformation unit 36 ​​generates the first, second, third, and fourth deformation gradient images Gd1, Gd2, Gd3, and Gd4 from the gradient image Gd of the infrared image Gb, based on the deformation parameter set (step S5).

[0104] The gradient synthesis unit 37 (first synthesis unit) synthesizes the gradient image Gc (second image) of the luminance image obtained by imaging with the visible light imaging device 14 with each of the multiple deformed gradient images to generate multiple composite gradient images (first composite image). In the example shown in Figure 12, the gradient synthesis unit 37 synthesizes the gradient image Gc with the first, second, third, and fourth deformed gradient images Gd1, Gd2, Gd3, and Gd4, respectively, to generate the first, second, third, and fourth composite gradient images Ge1, Ge2, Ge3, and Ge4 (step S6).

[0105] The gradient evaluation and parameter selection unit 38 (selection unit) selects a predetermined composite gradient image from a plurality of composite gradient images based on the gradient energy, which is the evaluation index. The gradient evaluation and parameter selection unit 38 selects the composite gradient image with the smallest gradient energy from the plurality of composite gradient images. In the example shown in Figure 12, the gradient evaluation and parameter selection unit 38 selects the third composite gradient image Ge3, which has the smallest gradient energy, from the first, second, third, and fourth composite gradient images Ge1, Ge2, Ge3, and Ge4 (step S7).

[0106] The gradient evaluation and parameter selection unit 38 outputs the deformation parameters used to generate the composite gradient image selected in step S7 as modified deformation parameters. In the example shown in Figure 12, the gradient evaluation and parameter selection unit 38 outputs the deformation parameters used to generate the third composite gradient image Ge3 as modified deformation parameters (step S8).

[0107] The image deformation unit 31 (second deformation unit) deforms the infrared image Gb based on the modified deformation parameters to generate a deformed infrared image (second deformed image) (step S9).

[0108] The image synthesis unit 32 (second synthesis unit) synthesizes the deformed infrared image generated in step S9 with the luminance image (second image) obtained by imaging by the visible light imaging device 14 to generate a synthesized luminance image (second synthesized image) (step S10).

[0109] The luminance and color information synthesis unit 33 generates a synthesized luminance image and color information to produce a synthesized image Gf (step S11).

[0110] The display control unit 28 causes the composite image Gf (second composite image) to be displayed on the display device 5 as the display image (step S12).

[0111] The display control unit 28 determines whether or not to terminate the display of the display image (step S13).

[0112] If it is determined in step S13 that the display should not be terminated (step S13: No), the process returns to step S2. The processes from step S2 to step S12 are performed at a predetermined interval.

[0113] If it is determined in step S13 that the display should be terminated (step S13: Yes), the display of the image is terminated.

[0114] [Computer System] Figure 14 is a block diagram showing a computer system 1000 according to an embodiment. The control device 6 described above includes the computer system 1000. The computer system 1000 has a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 1003, and an interface 1004 including input / output circuits. The functions of the control device 6 described above are stored in the storage 1003 as a computer program. The processor 1001 reads the computer program from the storage 1003, loads it into the main memory 1002, and executes the above-described processing according to the program. The computer program may be distributed to the computer system 1000 via a network.

[0115] The computer program or computer system 1000 can perform the following actions according to the above-described embodiment: generate a plurality of deformed gradient images from a gradient image Gd obtained by imaging with the infrared imaging device 15; generate a plurality of composite gradient images by combining the gradient image Gc obtained by imaging with the visible light imaging device 14 with each of the plurality of deformed gradient images; select a predetermined composite gradient image from the plurality of composite gradient images based on the gradient energy; and display the composite image Gf generated based on the selected composite gradient image as a display image on the display device 5.

[0116] [effect] As described above, according to the embodiment, the display system 18 includes an image deformation unit 36 ​​that generates a plurality of deformation gradient images (first, second, third, and fourth deformation gradient images Gd1, Gd2, Gd3, and Gd4 in the example shown in Figure 12) from a gradient image Gd obtained by imaging with an infrared imaging device 15; a gradient synthesis unit 37 that generates a plurality of composite gradient images (first, second, third, and fourth composite gradient images Ge1, Ge2, Ge3, and Ge4 in the example shown in Figure 12) by combining a gradient image Gc obtained by imaging with a visible light imaging device 14 and each of the plurality of deformation gradient images; a gradient evaluation / parameter selection unit 38 that selects a predetermined composite gradient image (third composite gradient image Ge3 in the example shown in Figure 12) from the plurality of composite gradient images based on the gradient energy; and a display control unit 28 that displays a display image generated based on the selected composite gradient image on the display device 5. As a result, even if the visible light image Ga captured by the visible light imaging device 14 becomes unclear, the unclear parts of the visible light image Ga are compensated for by the infrared image Gb, and the occurrence of ghost edges in the displayed image is suppressed. Even if the composite image of the visible light image Ga and the infrared image Gb becomes unclear, the display system 18 can properly provide the operator of the work machine 1 with information about the surrounding conditions of the work machine 1.

[0117] The image deformation unit 36 ​​shifts the gradient image Gd in a predetermined shift direction (left-right direction in this embodiment) to generate multiple gradient deformation images. Due to the parallax between the visible light imaging device 14 and the infrared imaging device 15 with respect to the object being imaged, simply combining the gradient image Gc and the gradient image Gd may result in ghost edges occurring in the combined image Gf. In this embodiment, the gradient image Gd and the gradient image Gc, which have been shifted in a predetermined shift direction, are combined, thus suppressing the occurrence of ghost edges.

[0118] The image deformation unit 36 ​​generates multiple gradient-deformed images from a gradient image Gd based on a deformation parameter set that includes multiple deformation parameters. The deformation parameter set is predetermined according to the conditions of the work site. The deformation parameters include the amount of shift of the gradient image Gd in a predetermined shift direction. The image deformation unit 36 ​​can efficiently generate multiple gradient-deformed images from the gradient image Gd based on the deformation parameter set.

[0119] As explained with reference to Figure 12, in this embodiment, when a composite gradient image is generated, the position of the gradient image Gc generated from the visible light image Ga is fixed, and the gradient image Gd generated from the infrared image Gb is shifted in the screen direction. When no event occurs that causes the visible light image Ga to become blurry, the visible light image Ga is displayed on the display device 5, and the operator operates the remote control device 4 while checking the visible light image Ga. When an event occurs that causes the visible light image Ga to become blurry, if the gradient image Gc generated from the visible light image Ga is shifted in the screen direction, the position of the visible light image Ga on the display device 5 may shift depending on whether the event occurs or not. If the position of the visible light image Ga shifts, the operator may feel uncomfortable. By fixing the position of the gradient image Gc generated from the visible light image Ga and shifting the gradient image Gd generated from the infrared image Gb in the screen direction, the shift in the position of the visible light image Ga on the display device 5 depending on whether the event occurs or not is suppressed.

[0120] [Other embodiments] As explained with reference to Figure 12, in the above embodiment, when generating a composite gradient image, the position of the gradient image Gc generated from the visible light image Ga is fixed, and the gradient image Gd generated from the infrared image Gb is shifted in the screen direction. Alternatively, the position of the gradient image Gd generated from the infrared image Gb may be fixed, and the gradient image Gc generated from the visible light image Ga may be shifted in the screen direction. Furthermore, both the gradient image Gc and the gradient image Gd may be shifted in the screen direction.

[0121] In the above-described embodiment, the visible light imaging device 14 and the infrared imaging device 15 are arranged in the left-right direction. The visible light imaging device 14 and the infrared imaging device 15 may also be arranged in the up-down direction. In step S5 shown in Figure 13, the image deformation unit 36 ​​may shift the gradient image Gd in the up-down direction.

[0122] In the above-described embodiment, when the gradient evaluation / parameter selection unit 38 selects a predetermined composite gradient image Ge from a plurality of composite gradient images Ge, gradient energy is used as the evaluation index. The gradient evaluation / parameter selection unit 38 may also select the composite gradient image Ge with the smallest gradient energy across the entire captured image. Furthermore, the gradient evaluation / parameter selection unit 38 may, for example, generate a composite gradient image Ge with low gradient energy in an image region arbitrarily selected by the operator on the display image, and display the area around the selected image region as clearly as possible (without image shift).

[0123] In the above-described embodiment, the phenomenon causing the image captured by the visible light imaging device 14 to become unclear was assumed to be the generation of dust. Other examples of phenomena causing the image captured by the visible light imaging device 14 to become unclear include the generation of fog, insufficient visible light due to the operation of the work machinery at night, and the subject being backlit.

[0124] When fog particles (water droplets) are present in the space between the visible light imaging device 14 and the infrared imaging device 15 and the object to be imaged, visible light has difficulty penetrating the fog particles, but infrared light can penetrate them. Therefore, the display system 18 can generate a composite image Gf according to the above-described embodiment. Even if at least a portion of the visible light image Ga captured by the visible light imaging device 14 becomes unclear, the display system 18 can provide the operator of the work machine 1 with information about the surroundings of the work machine 1 by displaying the composite image Gf on the display device 5.

[0125] In the above-described embodiment, the display system 18 is applied to the remote control system 2. The display device 5 does not have to be located in the remote control room 3. The display device 5 may be located in the control room (cab) of the work machine 1. In addition, some of the functions of the control device 6 described in the above-described embodiment may be located in the work machine 1. An operator who is in the control room of the work machine 1 can operate the boarding control device located in the control room of the work machine 1 while checking the display device 5 located in the control room of the work machine 1. Even in this case, the display system 18 can provide the operator of the work machine 1 with information about the surroundings of the work machine 1, even if the visible light image Ga captured by the visible light imaging device 14 becomes unclear.

[0126] In the above-described embodiment, the work machine 1 is assumed to be a hydraulic excavator. The work machine 1 may also be a bulldozer, a wheel loader, or a dump truck. If the work machine 1 is, for example, a dump truck, then in addition to an image of the area around the work machine 1, an image of the direction in which the work machine 1 is moving may also be displayed on the display device 5 as an image of the work site. [Explanation of Symbols]

[0127] 1...Working machine, 2...Remote control system, 3...Remote control room, 4...Remote control device, 5...Display device, 6...Control device, 7...Operator's seat, 8...Control device, 9...Communication system, 10...Traveling body, 11...Slewing body, 12...Working machine, 12A...Boom, 12B...Arm, 12C...Bucket, 13...Hydraulic cylinder, 13A...Boom cylinder, 13B...Arm cylinder, 13C...Bucket cylinder, 14...Visible light imaging device, 14A...Optical system, 14B...Image sensor, 14C...Imaging surface, 15...Infrared imaging device, 15A...Optical system, 15B...Image sensor, 15C...Imaging surface, 16...Communication device, 17...Communication device, 18...Display system, 19...Traveling body control unit, 20...Slewing body control unit, 21...Working machine control unit, 22...Image output unit, 23... 1000...Operation signal output unit, 24...Visible light image acquisition unit, 25...Infrared image acquisition unit, 26...Deformation parameter acquisition unit, 27...Image processing unit, 28...Display control unit, 29...Brightness and color information separation unit, 30...Alignment processing unit, 31...Image deformation unit, 32...Image synthesis unit, 33...Brightness and color information synthesis unit, 34...Gradient information extraction unit, 35...Gradient information extraction unit, 36...Image deformation unit, 37...Gradient synthesis unit, 38...Gradient evaluation and parameter selection unit, 1000...Computer system, 1001...Processor, 1002...Main memory, 1003...Storage, 1004...Interface, Ga...Visible light image, Gb...Infrared image, Gc...Gradient image, Gd...Gradient image, Ge...Composite gradient image, Gf...Composite image, AX1...Optical axis, AX2...Optical axis, RX...Swivel axis.

Claims

1. A first deformation unit generates a plurality of first deformation images from a first image of a first imaging range obtained by imaging with a first imaging device, A first combining unit generates a plurality of first combined images by combining a second image of a second imaging range that is not captured by imaging by the second imaging device and which coincides with the first imaging range, with each of the plurality of first modified images. A selection unit that selects a predetermined first composite image from a plurality of the first composite images, The system includes a display control unit that causes a display image generated based on the selected first composite image to be displayed on a display device, Generating the first deformed image from the first image includes shifting the first image in a predetermined shift direction. The first imaging device and the second imaging device are arranged apart in the shift direction. The first deformation unit shifts the first image by a different amount at each of the multiple positions in the depth direction parallel to the optical axis of the first imaging device, to each of the multiple positions in the screen direction intersecting the optical axis, thereby generating a plurality of first deformation images. Display system.

2. The first deformation unit deforms the first image based on each of a plurality of deformation parameters to generate a plurality of first deformed images. The display system according to claim 1.

3. The selection unit outputs the selected deformation parameters used to generate the first composite image as modified deformation parameters. A second deformation unit that deforms the first image based on the aforementioned modification deformation parameters to generate a second deformed image, The system includes a second synthesis unit that synthesizes the second deformed image and the second image to generate a second synthesized image, The displayed image includes the second composite image, The display system according to claim 2.

4. The first imaging device is an infrared imaging device, The previous second imaging device is a visible light imaging device. The display system according to any one of claims 1 to 3.

5. The first imaging device and the second imaging device are arranged adjacent to each other in the work machine. The display system according to any one of claims 1 to 4.

6. The first and second images are gradient images from which the edges of the object being captured have been extracted. The display system according to any one of claims 1 to 5.

7. The selection unit selects the first composite image in which the proportion of the edges is smallest as the predetermined first composite image. The display system according to claim 6.

8. The first imaging device and the second imaging device have parallel optical axes and their imaging ranges overlap in at least a portion of each other. The display system according to any one of claims 1 to 7.

9. The first imaging device is an infrared imaging device, The previous second imaging device is a visible light imaging device, The second deformation unit deforms the infrared image captured by the infrared imaging device based on the modified deformation parameters to generate the second deformed image. The display system according to claim 3.

10. To generate multiple first deformed images from the first image of the first imaging range obtained by imaging with the first imaging device, A portion of the object to be imaged obtained by imaging with the second imaging device is not captured, and a second image of the second imaging range that coincides with the first imaging range is combined with each of the multiple first modified images to generate a plurality of first composite images. Selecting a predetermined first composite image from a plurality of the first composite images, This includes displaying a display image generated based on the selected first composite image on a display device, Generating the first deformed image from the first image includes shifting the first image in a predetermined shift direction. The first imaging device and the second imaging device are arranged apart in the shift direction. At each of the multiple positions in the depth direction parallel to the optical axis of the first imaging device, the first image is shifted by a different amount to each of the multiple positions in the screen direction intersecting the optical axis, thereby generating a plurality of the first deformed images. Display method.

Images