Display control device and display control method

Abstract

The present invention provides a display control device and a display control method that enable the use of information obtained from previously displayed images as much as possible when displaying images representing other information on a display. [Solution] The display control device is a display control device that displays information relating to a work machine on a display screen, wherein the display screen includes an image display area that displays an image captured using a camera installed on the work machine to capture an image of the area around the work machine, and when predetermined information to be notified using the display screen is generated, a pop-up window smaller in size than the image display area and containing the predetermined information is displayed in a predetermined area including the image display area.

Description

Technical Field

[0003]

[0001] The present disclosure relates to a display control device and a display control method.

Background Art

[0002] Patent Document 1 describes a display system that displays, as a default display image, a display image including an overhead image representing the surroundings of a work machine generated based on captured images captured by a plurality of cameras and a single-camera image captured by one camera on a display provided inside the driver's cab of the work machine.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When an operator is performing normal work, the default display image of Patent Document 1 is displayed. When a guidance message such as a fixed playback process is notified during normal work, a part of the overhead image or the single-camera image displayed on the display is hidden. In this case, there is a problem that the operator cannot confirm the overhead image or the single-camera image.

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to provide a display control device and a display control method that enable information obtained from an image that has been displayed so far to be used as much as possible when an image representing other information such as a guidance message is displayed on a display. [[ID=​​​​​The display control device of the present disclosure is a display control device that displays information relating to a work machine on a display screen, wherein the display screen includes an image display area that displays an image captured using a camera installed on the work machine to capture an image of the area around the work machine, and when predetermined information to be notified using the display screen is generated, a pop-up window smaller in size than the image display area and containing the predetermined information is displayed in a predetermined area including the image display area.

[0007] The display control method of the present disclosure is a display control method for displaying information relating to a work machine on a display screen, wherein the display screen includes an image display area that displays an image captured using a camera installed on the work machine to capture an image of the area around the work machine, and when predetermined information to be notified using the display screen is generated, a pop-up window smaller in size than the image display area and containing the predetermined information is displayed in a predetermined area including the image display area. [Effects of the Invention]

[0008] According to the display control device and display control method of this disclosure, when displaying an image representing other information such as a guidance message on a display (display screen), it is possible to utilize as much information as possible that can be obtained from the previously displayed image. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing the configuration of a work machine according to an embodiment of this disclosure. [Figure 2] This figure shows the internal configuration of the driver's cab according to the embodiment of this disclosure. [Figure 3] This is a schematic block diagram showing the hardware configuration of a control system according to an embodiment of the present disclosure. [Figure 4] This figure shows an example of a display screen according to an embodiment of this disclosure. [Figure 5] This figure shows an example of a display screen according to an embodiment of this disclosure. [Figure 6]This figure shows an example of a display screen according to an embodiment of this disclosure. [Figure 7] This figure shows an example of a display screen according to an embodiment of this disclosure. [Figure 8] This figure shows an example of a display screen according to an embodiment of this disclosure. [Figure 9] This flowchart shows an example of the operation of the display control unit according to the embodiment of this disclosure. [Figure 10] This figure shows another example of a display screen according to the embodiments of this disclosure. [Figure 11] This flowchart shows another example of operation of the display control unit according to the embodiment of this disclosure. [Modes for carrying out the invention]

[0010] Embodiments of this disclosure will be described below with reference to the drawings. In each drawing, the same or corresponding components are given the same reference numerals, and their descriptions will be omitted as appropriate.

[0011] (Configuration of the work machine 100) Figure 1 is a schematic diagram showing the configuration of a work machine 100 according to an embodiment of the present disclosure. The work machine 100 operates at a construction site and performs construction work on materials such as soil and sand. The work machine 100 according to an embodiment of the present disclosure is a hydraulic excavator. However, the work machine of the present disclosure may be other work machines such as a dump truck, wheel loader, or motor grader. The work machine 100 comprises a traveling body 110, a rotating body 120, a work machine 130, and an operator's cab 140.

[0012] The traveling body 110 supports the work machine 100 so that it can move. The traveling body 110 is equipped with two tracks 111a and 111b on the left and right sides, and two travel motors 112 for driving each track 111a and 111b. The slewing body 120 is supported by the traveling body 110 so that it can slewing around a pivot point.

[0013] The working machine 130 is hydraulically driven. The working machine 130 is supported at the front part of the revolving body 120 so as to be vertically drivable. The cab 140 is a space for an operator to board and operate the working machine 100. The cab 140 is provided at the left front part of the revolving body 120. In this embodiment, as shown in FIG. 1, the vertical, left - right, and front - rear directions are defined based on the revolving body 120. Also, the part of the revolving body 120 to which the working machine 130 is attached is referred to as the front part. Further, for the revolving body 120, the part on the opposite side with respect to the front part is referred to as the rear part, the part on the left side is referred to as the left part, and the part on the right side is referred to as the right part.

[0014] (Configuration of the revolving body 120) The revolving body 120 includes an engine 121, a hydraulic pump 122, a control valve 123, a slewing motor 124, a fuel injection device 125, an exhaust gas purification device 127, etc. The engine 121 is a prime mover that drives the hydraulic pump 122. The engine 121 is an example of a power source. In this embodiment, the engine 121 is a diesel engine.

[0015] The hydraulic pump 122 is a variable displacement pump driven by the engine 121. The hydraulic pump 122 supplies hydraulic oil to a plurality of actuators via the control valve 123. The plurality of actuators includes a boom cylinder 131C, an arm cylinder 132C, a bucket cylinder 133C, a traveling motor 112, and a slewing motor 124. The control valve 123 controls the flow rate of the hydraulic oil supplied from the hydraulic pump 122.

[0016] The swing motor 124 is driven by hydraulic oil supplied from the hydraulic pump 122 via the control valve 123 to swing the swing body 120. The fuel injection device 125 injects fuel into the engine 121. The exhaust gas purification device 127 is a device that purifies nitrogen oxides (NOx) and particulate matter (PM (Particulate Matter)) contained in the exhaust gas of the engine 121. The exhaust gas purification device 127 includes a DPF device, an SCR device, etc. provided in the exhaust passage of the engine 121. The DPF device includes a DOC (Diesel Oxidation Catalyst) and a DPF (Diesel Particulate Filter), collects PM with the DPF, oxidizes the PM collected downstream by nitrogen dioxide converted by the DOC into carbon dioxide, and removes the PM. The SCR device includes an SCR (Selective Catalytic Ruduction; ammonia selective reduction type catalyst), an injector, a pump, and a tank for supplying aqueous urea solution into the exhaust gas upstream of the SCR, and converts nitrogen oxides (NOx) into nitrogen molecules (N2) and water (H2O).

[0017] (Configuration of the working machine 130) The working machine 130 includes a boom 131, an arm 132, a bucket 133, a boom cylinder 131C, an arm cylinder 132C, and a bucket cylinder 133C.

[0018] The base end portion of the boom 131 is attached to the swing body 120 via a boom pin P1. The arm 132 connects the boom 131 and the bucket 133. The base end portion of the arm 132 is attached to the tip end portion of the boom 131 via an arm pin P2. The bucket 133 includes a blade for excavating earth and sand and a storage portion for storing the excavated earth and sand. The base end portion of the bucket 133 is attached to the tip end portion of the arm 132 via a bucket pin P3.

[0019] The boom cylinder 131C is a hydraulic cylinder for driving the boom 131. The base end of the boom cylinder 131C is attached to the slewing body 120. The tip end of the boom cylinder 131C is attached to the boom 131. The arm cylinder 132C is a hydraulic cylinder for driving the arm 132. The base end of the arm cylinder 132C is attached to the boom 131. The tip end of the arm cylinder 132C is attached to the arm 132. The bucket cylinder 133C is a hydraulic cylinder for driving the bucket 133. The base end of the bucket cylinder 133C is attached to the arm 132. The tip end of the bucket cylinder 133C is attached to the link member connected to the bucket 133.

[0020] (Example camera configuration) The rotating body 120 is equipped with multiple cameras 126 (front camera 126A, left-side camera 126B, rear camera 126C, and right-side camera 126D) that capture images of the area around the work machine 100. The front camera 126A is located inside the operator's cab 140. The front camera 126A, left-side camera 126B, rear camera 126C, and right-side camera 126D are collectively referred to as multiple cameras 126. Alternatively, instead of multiple cameras 126, or in conjunction with multiple cameras 126, a single omnidirectional camera may be used. The display unit 206, described later, displays overhead images and single-camera images based on the images (single-camera images) captured by each camera 126.

[0021] (Configuration of the 140 driver's cab) Figure 2 shows the internal configuration of the driver's cab 140 according to the embodiment of this disclosure. The driver's cab 140 is equipped with a driver's seat 142, an operating device 143, a display unit 206, and the like.

[0022] The operating device 143 is a device for driving the traveling body 110, the slewing body 120, and the work equipment 130 by manual operation by the operator. The operating device 143 comprises a left operating lever 143LO, a right operating lever 143RO, a left foot pedal 143LF, a right foot pedal 143RF, a left traveling lever 143LT, and a right traveling lever 143RT. The left operating lever 143LO is located on the left side of the driver's seat 142. The right operating lever 143RO is located on the right side of the driver's seat 142.

[0023] The left operating lever 143LO is an operating mechanism for, for example, the rotational movement of the slewing body 120 and the digging / dumping movement of the arm 132. The right operating lever 143RO is an operating mechanism for, for example, the digging / dumping movement of the bucket 133 and the raising / lowering movement of the boom 131.

[0024] The left foot pedal 143LF is located on the left side of the floor in front of the driver's seat 142. The right foot pedal 143RF is located on the right side of the floor in front of the driver's seat 142. The left travel lever 143LT is pivotally supported by the left foot pedal 143LF, and the tilt of the left travel lever 143LT is linked to the downward movement of the left foot pedal 143LF. The right travel lever 143RT is pivotally supported by the right foot pedal 143RF, and the tilt of the right travel lever 143RT is linked to the downward movement of the right foot pedal 143RF.

[0025] The left foot pedal 143LF and the left travel lever 143LT correspond to the rotational drive of the left track 111a of the vehicle body 110. The right foot pedal 143RF and the right travel lever 143RT correspond to the rotational drive of the right track 111b of the vehicle body 110.

[0026] The display unit 206 is, for example, a touch panel. The display unit 206 is an input / output device that integrally comprises an input unit 206I (Figure 3) that accepts input operations by touching the display screen 206S with a fingertip or an input pen, and an output device that displays an image on the display screen 206S. The display unit 206 is controlled by a display control unit 2012, which will be described later, and displays predetermined information representing the status of the work machine 100 on the display screen 206S. The predetermined information representing the status of the work machine 100 that the display screen 206S displays is, for example, information representing the operating status of the work machine 100, information representing the setting status and operation status, image information representing the surroundings of the work machine 100 based on moving images, still images, etc., captured by multiple cameras 126. For example, predetermined information representing the status of the work machine 100 is also called information related to the work machine 100. In this embodiment, "information" refers to the content that is transmitted, and includes image information that represents the content as an image, numerical information that represents the content as a number, etc. Furthermore, in this embodiment, the vertical and horizontal directions of the display screen 206S are as shown in Figure 2. Also, in this embodiment, the display screen 206S has a rectangular shape with its longitudinal direction being vertical. The display unit 206 receives input operations for the display screen 206S and outputs information indicating the operation content to the display control unit 2012, which will be described later. Alternatively, the display unit 206 may also receive input operations for the display screen 206S and output information indicating the operation content to the work machine control unit 2013, which will be described later.

[0027] (Configuration of control system 200) Figure 3 is a schematic block diagram showing the hardware configuration of the control system 200 of the work machine 100 according to an embodiment of the present disclosure. The control system 200 includes a controller 201. The controller 201 includes an input / output unit 2011, a display control unit 2012, and a work machine control unit 2013. The control system 200 also includes a plurality of cameras 126, a display unit 206, a plurality of sensors 208, a plurality of actuators 209, etc. In this embodiment, the video signal is also referred to as image information. Furthermore, the controller 201, the display control unit 2012, or the display image generation unit 2063 are examples of the configuration of the "display control device" in this disclosure.

[0028] The controller 201 and the display control unit 2012 can be configured using a computer such as a microcontroller or CPU (Central Processing Unit), and hardware such as peripheral circuits and peripheral devices of the computer. The controller 201 and the display control unit 2012 may also be configured using a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device). Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). Furthermore, some or all of the functions realized by the processor may be realized by the integrated circuit. The controller 201 is a functional block composed of a combination of hardware such as a computer and software such as a program executed by the computer, and includes an input / output unit 2011, a display control unit 2012, and a work machine control unit 2013. The controller 201 may be configured as a single controller, or it may be configured as multiple controllers connected and cooperating via a predetermined communication line, for example.

[0029] The input / output unit 2011 inputs and outputs predetermined signals (input or output or input and output) to and from multiple cameras 126, multiple sensors 208, and multiple actuators 209.

[0030] The work machine control unit 2013 acquires various data related to the hydraulic equipment that controls the operation of the work machine 130 using multiple sensors 208, and outputs multiple control signals to multiple actuators 209 related to the work machine 130 to control the hydraulic equipment according to the operation of the operating device 143. In other words, the work machine control unit 2013 controls the drive of the boom cylinder 131C, arm cylinder 132C, bucket cylinder 133C, travel motor 112, slewing motor 124, etc. The work machine control unit 2013 is an example of a vehicle body control unit that outputs control signals to drive the vehicle body of the work machine 100 with power supplied by the power source. The work machine control unit 2013 also transmits status information related to the operation of the work machine 100 to the display control unit 2012. The work machine control unit 2013 also transmits predetermined information representing the state of the work machine 100 (hereinafter also referred to as status information) to the display control unit 2012, etc.

[0031] In this embodiment, the predetermined information (state information) representing the state of the work machine 100 includes image information representing images captured by multiple cameras 126 and image information such as overhead images generated by image processing from the captured images, information representing measurement results from multiple sensors 208, numerical information obtained based on information representing measurement results from multiple sensors 208 and control commands to multiple actuators 209, notification information to notify the operator and warning information to draw attention to the operator. The multiple sensors 208 include a sensor for detecting the hydraulic oil temperature, a sensor for detecting the hydraulic pressure, a sensor for detecting the operating state of the operating device 143, a sensor for detecting the posture of the work machine 130, etc. The multiple actuators 209 include a hydraulic pump 122, a control valve 123, a boom cylinder 131C, an arm cylinder 132C, a bucket cylinder 133C, a travel motor 112, a slewing motor 124, etc.

[0032] Furthermore, the work machine control unit 2013 acquires various data related to the engine 121, etc., using multiple sensors 208, and outputs multiple control signals for controlling the engine 121, etc., to multiple actuators 209 related to the engine 121. The work machine control unit 2013 also transmits information related to the engine 121 to the display control unit 2012, etc. The information related to the engine 121 is an example of status information of the work machine 100. The multiple sensors 208 include a sensor for detecting engine speed, a sensor for detecting engine coolant temperature, a sensor for detecting fuel level, a sensor for detecting urea solution level, etc. The multiple actuators 209 include a fuel injector 125, a urea solution injector, a pump, etc. Note that the connection configuration between the multiple sensors 208 and actuators 209 and the work machine control unit 2013 shown in Figure 3 is just one example, and the connection configuration is not limited to the example shown in Figure 3.

[0033] Furthermore, the work machine control unit 2013 controls the regeneration of the exhaust gas purification device 127. In addition to NOx and PM, diesel engine exhaust also contains HC (hydrocarbons), and in the low-load operating range of the engine, this HC accumulates in the DOC, DPF, and SCR. If this condition continues for a long time, the amount of HC accumulation will continue to increase, and if the engine 121 is then operated under high load, the temperature inside the exhaust gas purification device 127 will rise, causing the accumulated HC to burn rapidly and potentially damaging the exhaust gas purification device 127. For this reason, when the engine operating conditions remain in the low-load range for a long time, and it is determined that a certain amount of HC has accumulated to the point where there is no risk of damage, the exhaust temperature is periodically increased to release the accumulated HC. The work machine control unit 2013 controls, for example, post-injection to raise the exhaust temperature and performs automatic control such as HC emission control for accumulated HC in the exhaust gas purification device 127, or control such as HC emission control after fixing the engine speed to a certain speed based on manual instructions (hereinafter, this control is called stationary manual regeneration). Stationary manual regeneration is a control that, with the operator's permission, stops normal operation and restores the performance of the exhaust gas purification device 127 when the exhaust temperature does not rise sufficiently in the normal operating state (a state in which normal work can be performed without fixing the engine speed to a certain speed), and PM, urea deposit removal and HC emission cannot be performed. In stationary manual regeneration, the work machine control unit 2013 first uses the display unit 206 to inform the operator that it is possible to perform stationary manual regeneration and to encourage them to do so. In response, when the operator uses the display unit 206 to instruct the machine to perform stationary manual regeneration, the machine control unit 2013 fixes the engine speed to a certain speed, increases the exhaust temperature, and removes PM or urea deposits and releases HC.

[0034] The display control unit 2012 repeatedly receives each video signal representing each image captured by the multiple cameras 126 at a predetermined period, and generates an overhead view image that displays the area around the work machine 100 as if viewed from above, based on each image captured by the multiple cameras 126. The display control unit 2012 outputs the video signal representing the generated overhead view image to the display unit 206, or outputs a video signal representing an image captured by any of the multiple cameras 126 (hereinafter also referred to as a single-camera image) to the display unit 206. In this embodiment, a single-camera image is an image captured using a camera 126 installed on the work machine 100, capturing a predetermined direction around the work machine 100.

[0035] Furthermore, the display control unit 2012 controls the image display on the display screen 206S of the display unit 206, and notifies the work machine control unit 2013, etc., of the occurrence of a touch operation on the input unit 206I of the display unit 206. In this embodiment, the display control unit 2012 comprises an information acquisition unit 2061, an input unit 2062, a display image generation unit 2063, and a display screen control unit 2064 as a functional block composed of a combination of hardware such as a computer and software such as a program executed by the computer.

[0036] The information acquisition unit 2061 acquires image information and other data, as well as status information of the work machine 100, including information indicating that manual regeneration should be performed, and information indicating the setting status of the work machine 100, such as the operating mode, from the work machine control unit 2013.

[0037] The input unit 2062 receives signals from the display unit 206 that represent input operations to the input unit 206I of the display unit 206, outputs them to the display image generation unit 2063, etc., and notifies the work machine control unit 2013, etc., of a signal indicating that a predetermined input operation has been performed. In other words, the input unit 2062 receives input from the operator to the display unit 206.

[0038] The display image generation unit 2063 generates an image to be displayed on the display screen 206S based on a predetermined set of display modes, using information acquired by the information acquisition unit 2061, signals input by the input unit 2062, etc. In this embodiment, "display screen" (display screen 206S) refers to the surface on which the image is projected. "Display mode" refers to the shape in which the image is projected onto the display screen 206S, for example, what kind of information is displayed, where on the display screen 206S, and how. The image projected onto the display screen 206S is determined by the display mode and the content of the information to be displayed. The display image generation unit 2063 also changes the display mode based on information acquired by the information acquisition unit 2061, signals input by the input unit 2062 (such as signals indicating input operations to the input unit 206I), etc.

[0039] The display screen control unit 2064 controls the display unit 206 to display the image generated by the display image generation unit 2063 on the display screen 206S.

[0040] Here, with reference to Figures 4 to 8, an example of the display of an image generated by the display image generation unit 2063 on the display screen 206S will be described. Figures 4 and 5 show an example of the display using the same display mode D1. Display mode D1 is the display mode that is displayed on the display screen 206S by default when the work machine 100 is in an operational state. Note that Figure 4 shows an example of the display in this embodiment, and Figure 5 shows multiple display areas DA1 and DA2 included in the display screen 206S with thick dashed or chained lines added for explanatory purposes. Display mode D1 includes display area DA1 and display area DA2. In display mode D1, display area DA1 and display area DA2 are adjacent to each other, and are arranged from top to bottom in the order of display area DA1 and display area DA2.

[0041] As shown in Figures 4 and 5, in display mode D1, the overhead image IM1 is displayed in display area DA1 and the single-camera image IM2 is displayed in display area DA2. Display mode D1 also includes a gauge EG1 that indicates the status of carbon dioxide emissions and fuel consumption. For example, the gauge EG1 may be omitted. The overhead image IM1 and the single-camera image IM2 correspond to the "information relating to the work machine" in this disclosure. Furthermore, the display area DA1 that displays the overhead image IM1 corresponds to the "overhead image display area" in this disclosure, and the display area DA2 that displays the single-camera image IM2 corresponds to the "captured image display area" in this disclosure. Note that the single-camera image IM2 shown in Figures 4 and 5 is the captured image from the rear camera 126C. Furthermore, the captured image of the single-camera image IM2 can be classified into an area RN1 close to the work machine 100 and an area RF1 far from the work machine 100. The close area RN1 is of higher importance to monitor than the far area RF1. In other words, the area RN1 close to the work machine 100 is the vicinity of the work machine 100. The nearby region RN1 and the distant region RF1 can be classified for each camera 126 during the design phase of the control system 200, depending on, for example, the shape of the work machine 100, the specifications of the camera 126, its installation position, and imaging direction.

[0042] Figure 6 shows an example of the display on the display screen 206S in display mode D2 when the information acquisition unit 2061 acquires information indicating a recommendation to perform stationary manual regeneration, while the display is being performed in display mode D1 as shown in Figures 4 and 5. In display mode D2 shown in Figure 6, a pop-up window PU1 is displayed on a predetermined area DA3 of the display screen 206S, which does not include the display area DA1 but includes the display area DA2. The pop-up window PU1 is smaller in size (area, perimeter, etc.) than the display area D2 and contains information indicating a recommendation to perform stationary manual regeneration. Area DA3 includes a part of the distant area RF1 within the display area DA2 and a part of the display area of ​​the gauge EG1. Area DA3 is also set so as not to overlap with the nearby area RN1. In other words, the pop-up window PU1 is not displayed in area RN1, which is close to the work machine 100. By not displaying the pop-up window PU1 in area RN1, which is close to the work machine 100, it is possible to check for people or objects in area RN1, which is close to the work machine 100. Pop-up window PU1 is a child window displayed superimposed on the currently displayed image. In the example shown in Figure 6, it includes an icon PU11, the text string PU12 which reads "We recommend performing stationary manual playback," and a button PU13 for closing pop-up window PU1. It is desirable that icon PU11 be the same as the icon used on the menu screen for performing stationary manual playback (not shown). In other words, by displaying the same icon as the one used on the menu screen, it is possible to encourage the operator to perform stationary manual playback. For example, the comment displayed on text string PU12 can be set to any text string. The information representing the guidance recommending that stationary manual playback be performed (hereinafter also referred to as guidance information), icon PU11, and text string PU12 correspond to "guidance information notifying prescribed matters" in this disclosure. Note that "guidance information" is included in "information related to the work machine." Also, area DA3 corresponds to "a prescribed area including the captured image display area" in this disclosure. Note that pop-up window PU1 is automatically closed (cleared) for example when button PU13 is tapped, or when a certain amount of time has elapsed since pop-up window PU1 was displayed.As shown in Figure 5, in display mode D1, display area DA1 and display area DA2 are adjacent. However, for example, a partial area may be provided between display area DA1 and display area DA2, and the pop-up window PU1 may be displayed on the partial area.

[0043] The display mode of the pop-up window PU1 (display position, window size, icon type, text content, close button specifications, etc.) is determined by the display image generation unit 2063 according to the content of the information to be displayed in the pop-up and the content of the display area DA2 (for example, which camera 126's image is being captured). For example, the pop-up window PU2 is not displayed near the work machine 100 in the single camera image IM2 displayed in the display area DA2.

[0044] Figure 7 also shows an example of display using display mode D3. Display mode D3 differs from display mode D1, which was explained with reference to Figures 4 and 5, in that the single-camera image displayed in display area DA2 is different. In display mode D3, the single-camera image IM2a displayed in display area DA2 is an image captured by the front camera 126A. The single-camera image IM2a can be classified into an area RN2 close to the work machine 100 and an area RF2 far from it, for example, as shown in Figure 7. Furthermore, if the information acquisition unit 2061 acquires guidance information while displaying in display mode D3 as shown in Figure 7, the display screen 206S will be displayed using display mode D4 as shown in Figure 8, for example. In display mode D4, the guidance information is displayed in a pop-up window PU2 on an area DA4 that does not include display area DA1 but includes display area DA2, and does not overlap with the area RN2 close to the work machine 100. For example, the pop-up window PU2 may be displayed using the single-camera image IM2a so as not to overlap with the image IM3 of the work machine 100. The pop-up window PU2 is smaller than the display area DA2 and includes guidance information. The camera 126 that captures single-camera images is selected by the operator, for example, using a predetermined menu screen.

[0045] (Example of operation of display control unit 2012) Referring to the flowchart in Figure 9, an example of the processing flow when switching between display screen 206S of display mode D1 shown in Figures 4 and 5 and display screen 206S of display mode D2 shown in Figure 6 will be explained. The processing shown in Figure 9 is repeatedly executed by the display control unit 2012 at predetermined time intervals.

[0046] When the process shown in Figure 9 is initiated, first, the information acquisition unit 2061 of the display control unit 2012 acquires status information, etc. (but not image information) from the work machine control unit 2013, etc. (step S101). The "status information, etc." includes information representing the status of the work machine 100 (status information) and information representing control signals, etc. that are not included in the information representing the status of the work machine 100. The "status information, etc." acquired in step S101 may or may not include guidance information. In this case, the guidance information is information that indicates a recommendation for the operator to perform stationary manual regeneration. Guidance information is transmitted from the work machine control unit 2013, etc. to the display control unit 2012 when information to be notified arises. In other words, when information to be notified arises, it means that guidance information recommending that the operator perform stationary manual regeneration is sent. It is possible to configure the system so that a signal is sent when the information to be notified disappears, or so that the information to be notified is periodically sent from the work machine control unit 2013, etc., to the display control unit 2012 after it is generated until it disappears. In addition, in the process shown in Figure 9, the guidance information corresponds to the information to be displayed as a pop-up.

[0047] Next, the information acquisition unit 2061 acquires image information (video signal) from the camera 126 (step S102). Then, the display image generation unit 2063 updates the display content of the display screen 206S (step S103). In this embodiment, for example, the latest overhead image IM1 is displayed in the display area DA1, and the latest single-camera image IM2 is displayed in the display area DA2.

[0048] Next, the display image generation unit 2063 determines whether or not the pop-up window PU1 is currently being displayed (step S104). If the pop-up window PU1 is not currently being displayed (step S104: NO), the display image generation unit 2063 determines whether or not there is information that should be displayed as a pop-up (step S105). In this embodiment, the information that should be displayed as a pop-up is guidance information recommending that manual playback be performed. If there is information that should be displayed as a pop-up (step S105: YES), the display image generation unit 2063 determines whether or not the display conditions for the pop-up window PU1 are met (step S106). The display conditions for the pop-up window PU1 are the conditions for displaying the pop-up window PU1. For example, the conditions for displaying the pop-up window PU1 are met when the pop-up window PU1 is displayed for the first time after the control system 200 is started. For example, if the pop-up window PU1 is closed manually or automatically, and it is set to be automatically displayed again after a predetermined time, the conditions are met when that predetermined time has elapsed. For example, if the status information in step S101 includes guidance information recommending that manual playback be performed, the condition for displaying the pop-up window PU1 is met. For example, if there is no display condition, the process may proceed to step S107, which will be described later.

[0049] If the display conditions are met (step S106: YES), the display image generation unit 2063 determines the display mode of the pop-up window according to the content of the information to be displayed in the pop-up and the content of the display area DA2 (step S107). In step S106, for example, the latest overhead image IM1 is displayed in the display area DA1 and the latest single-camera image IM2 is displayed in the display area DA2. Next, the display image generation unit 2063 displays the pop-up window PU1 (step S108) and terminates the process shown in Figure 9.

[0050] If, as determined in step S105, there is no information to display in the popup (step S105: NO), and if, as determined in step S106, the display condition is not met (step S106: NO), the display image generation unit 2063 then terminates the process shown in Figure 9 without displaying the popup window PU1.

[0051] Furthermore, in the determination in step S104, if the popup window PU1 was being displayed (step S104: YES), the display image generation unit 2063 determines whether the display termination condition for popup window PU1 has been met (step S109). The display termination condition for popup window PU1 is the condition for ending the display of popup window PU1, and is met in the following cases, for example. For example, the condition for ending the display of popup window PU1 is met when the button PU13 for closing popup window PU1 is tapped. For example, the condition for ending the display of popup window PU1 may be met when a predetermined time has elapsed since the start of displaying popup window PU1.

[0052] If the display termination condition is met (step S109: YES), the display image generation unit 2063 terminates the display of the popup window PU1 (step S110) and ends the process shown in Figure 9. If the display termination condition is not met (step S109: NO), the display image generation unit 2063 terminates the process shown in Figure 9 without terminating the display of the popup window PU1.

[0053] For example, through the above processing, in this embodiment, the display screen 206S of display mode D1 shown in Figures 4 and 5 and the display screen 206S of display mode D2 shown in Figure 6 are switched and displayed. Note that the processing flow shown in Figure 9 is just one example, and the processing flow is not limited to the case shown in Figure 9. For example, processing such as updating the display content may be executed in parallel on an object-by-object basis.

[0054] (Other operation examples of the display control unit 2012) In the display examples of the display screen 206S shown in Figures 4 to 8, the display mode of the single camera image IM2 or IM2a displayed in the display area DA2 is the same whether or not the popup window PU1 or PU2 is displayed. In contrast, in other operation examples, as shown in Figure 10, in display mode D5 when the popup window PU1 is displayed, the single camera image IM2s displayed in the display area DA2 is scaled down so as not to overlap with the popup window PU1 and displayed in the display area DA2. Display mode D5 is the display mode when guidance information occurs when display mode D1 shown in Figure 4 is displayed. The processing flow in this case is as shown in Figure 11, and is the same as the flow in Figure 9, but with the processing of step S107-2 added between steps S107 and S108. In step S107-2, the display image generation unit 2063 changes the display mode of the display area DA2, and the information in the display area DA2 (single camera image IM2 in Figure 4) is scaled down to a single camera image IM2s, for example, without changing the aspect ratio so as not to overlap with the popup window PU1. In the flow shown in Figure 11, step S110 returns the display mode of the display area DA2 to the display mode when the pop-up window PU1 is not displayed.

[0055] (Effects / Actions) As described above, the display control unit 2012 (display image generation unit 2063) (display control device) of this embodiment displays information related to the work machine 100 on the display screen 206S. The display screen 206S includes a display area DA2 (image display area) that displays an image captured using a camera 126 installed on the work machine 100 to capture an image of the area around the work machine 100. When predetermined information (guidance information) to be notified using the display screen 206S is generated, the display control unit 2012 displays a pop-up window PU1 in a predetermined area (area DA3) that includes the display area DA2 (image display area), which is smaller in size than the display area DA2 (image display area) and contains the predetermined information (guidance information). With this configuration, when displaying an image representing other information (guidance information) on the display screen 206S, it is displayed using a pop-up window PU1 which is smaller in size than the display area DA2 (image display area), so that the information obtained from the previously displayed image (single camera image IM2) can be continuously used using a part of the display area DA2 (image display area) where the pop-up window PU1 is not displayed. Furthermore, even when displaying the pop-up window PU1, the overhead image IM1 can always be displayed in the display area DA1 (overhead image display area).

[0056] Furthermore, in this embodiment, when displaying the pop-up window PU1, it is displayed so as not to overlap with the image of the work machine 100 included in the captured image. The display screen 206S also includes a display area DA1 (overview image display area) that displays an overhead image of the area around the work machine 100, and the predetermined information is guidance information that notifies predetermined matters. Furthermore, when the captured image is classified into an area close to the work machine and an area far from the work machine, the pop-up window PU1 is displayed so as not to overlap with the area close to the work machine. The pop-up window PU1 also includes an icon corresponding to the content of the predetermined information. The pop-up window PU1 also includes a button for closing the pop-up window. The display screen 206S also includes a partial area located between the display area DA2 (captured image display area) and the display area DA1 (overview image display area), and the pop-up window PU1 is displayed on this partial area. These configurations provide the following effects.

[0057] In other words, the single-camera image IM2 is a single-camera image (captured image) taken using a camera 126 installed on the work machine 100 to capture a predetermined direction around the work machine 100. The display control unit 2012 displays the pop-up window PU1 so as not to overlap with the close region RN1 when the single-camera image IM2 (captured image) is classified into a region RN1 close to the work machine 100 and a region RF1 far away from the work machine 100. With this configuration, the pop-up window PU1 can be prevented from overlapping with regions of relatively high importance for monitoring.

[0058] Furthermore, the display screen 206S is a display area for predetermined information representing the status of the work machine 100, and may include a partial area located between the display area DA1 (overview image display area) and the display area DA2 (captured image display area). With this configuration, other information can be displayed between the display area DA1 (overview image display area) and the display area DA2 (captured image display area). In other words, guidance information, which is other information, may be displayed in the partial area.

[0059] Furthermore, when the display control unit 2012 displays the pop-up window PU1, it displays the camera image IM2s in a reduced size so as not to overlap with the pop-up window PU1. With this configuration, by reducing the size of the single camera image IM2s, the entire area of ​​the original single camera image IM2 can be displayed without overlapping with the pop-up window PU1.

[0060] Furthermore, when the display control unit 2012 displays the pop-up window PU2, it displays the pop-up window PU2 in such a way that it does not overlap with the image IM3 of the work machine 100 that is included in the single-camera image (captured image) IM2a. With this configuration, the pop-up window PU2 can be displayed without overlapping with the work machine 100.

[0061] Furthermore, the guidance information includes an icon PU11 corresponding to the content of the guidance information. With this configuration, for example, if the guidance information is information that recommends performing a fixed manual playback, the icon PU11 can be the same as the icon used on the menu screen for performing the fixed manual playback. In addition, the pop-up window PU1 includes a button for closing the pop-up window, making it easy to close the pop-up window PU1.

[0062] The display control method according to this embodiment is a display control method that displays information related to the work machine 100 on a display screen 206S, the display screen 206S includes a display area DA2 (image display area) that displays an image captured using a camera 126 installed on the work machine 100 to capture an image of the area around the work machine 100, and when predetermined information to be notified using the display screen 206S is generated, a pop-up window PU1 that is smaller in size than the display area DA2 (image display area) and contains the predetermined information is displayed in a predetermined area including the display area DA2 (image display area).

[0063] (Variations, etc.) While embodiments of this invention have been described above with reference to the drawings, the specific configuration is not limited to the embodiments described above, and design changes and the like that do not depart from the spirit of this invention are also included.

[0064] For example, the display unit 206 may use an input / output device equipped with a monitor without touch input functionality, such as an LCD (Liquid Crystal Display) or an organic electroluminescent display, and physical buttons, instead of a touch panel. Alternatively, the display screen 206S may be horizontally elongated, with display areas DA1 and DA2 arranged side-by-side. The prime mover may be an electric motor instead of an engine 121, or a combination of an engine and an electric motor. Furthermore, the display unit 206 may be located in a remote control room for remotely operating the work machine 100, instead of (or in addition to) the operator's cab 140 of the work machine 100. In this case, the operating device 143 and the display unit 206, etc., are configured to communicate with the work machine 100, enabling remote operation of the work machine 100. While the above embodiment displays a single-camera image IM2 in display area DA2, the information displayed in display area DA2 is not limited to this example. Furthermore, in the above embodiment, guidance information recommending the performance of stationary manual regeneration was given as an example of information to be displayed in the pop-up window PU1, but the invention is not limited to this example. For example, information notifying that the gauge EG1 has deviated from the recommended range may be displayed in the pop-up window.

[0065] Furthermore, some or all of the program executed by the computer in the above embodiment can be distributed via a computer-readable recording medium or communication line.

[0066] (Note) The display control unit 2012 (display control device) described in the embodiment can be understood as follows.

[0067] (1) A display control device according to a first aspect of the present disclosure is a display control device that displays information relating to a work machine on a display screen, wherein the display screen includes an image display area that displays an image captured using a camera installed on the work machine to capture an image of the area around the work machine, and when predetermined information to be notified using the display screen is generated, a pop-up window smaller in size than the image display area and containing the predetermined information is displayed in a predetermined area including the image display area.

[0068] (2) A display control device according to a second aspect of the present disclosure is the display control device according to (1), wherein when displaying the pop-up window, the pop-up window is displayed such that it does not overlap with the image of the work machine included in the captured image.

[0069] (3) A display control device according to a third aspect of the present disclosure is the display control device according to (1) or (2), wherein the display screen includes an overhead image display area for displaying an overhead image of the area around the work machine, and the predetermined information is guidance information for notifying predetermined matters.

[0070] (4) A display control device according to a fourth aspect of the present disclosure is the display control device according to (1) to (3), wherein the pop-up window is displayed such that it does not overlap with the area close to the work machine when the captured image is classified into an area close to the work machine and an area far from the work machine.

[0071] (5) A display control device according to a fifth aspect of the present disclosure is the display control device according to (1) to (4), wherein the pop-up window includes an icon corresponding to the content of the predetermined information.

[0072] (6) A display control device according to a sixth aspect of the present disclosure is the display control device according to (1) to (5), wherein the pop-up window includes a button for closing the pop-up window.

[0073] (7) A display control device according to a seventh aspect of the present disclosure is the display control device according to (1) to (5), wherein the display screen includes a partial area located between the captured image display area and the overhead image display area, and the pop-up window is displayed on the partial area. [Explanation of Symbols]

[0074] 100...Working machine 110...Traveling body 120...Slewing body 126, 126A~D...Camera 130...Working machine 140...Operator's cab 200...Control system 201...Controller 2012...Display control unit 2013...Working machine control unit 206...Display unit 206S...Display screen 206I...Input unit DA1, DA2...Display area DA3, DA4...Area

Claims

1. A display control device that displays information related to a work machine on a display screen, The display screen includes an image display area that displays an image captured using a camera installed on the work machine, which captures the area around the work machine. When predetermined information to be notified using the display screen is generated, a pop-up window smaller than the image display area and containing the predetermined information is displayed in a predetermined area including the image display area. Display control device.

2. When displaying the aforementioned pop-up window, the pop-up window is displayed in such a way that it does not overlap with the image of the work machine included in the captured image. The display control device according to claim 1.

3. The aforementioned display screen includes an overhead image display area that displays an overhead image of the area around the work machine, The aforementioned specified information is guidance information that notifies specified matters. The display control device according to claim 2.

4. When the captured image is classified into areas close to the work machine and areas far from the machine, the pop-up window is displayed so as not to overlap with the area close to the machine. The display control device according to claim 3.

5. The pop-up window includes an icon corresponding to the content of the predetermined information. The display control device according to claim 3.

6. The pop-up window includes a button for closing the pop-up window. A display control device according to any one of claims 1 to 5.

7. The aforementioned display screen includes a partial region located between the captured image display area and the overhead image display area. Display the aforementioned pop-up window on the aforementioned partial area. The display control device according to claim 3.

8. A display control method for displaying information related to a work machine on a display screen, The display screen includes an image display area that displays an image captured using a camera installed on the work machine, which captures the area around the work machine. When predetermined information to be notified using the display screen is generated, a pop-up window smaller than the image display area and containing the predetermined information is displayed in a predetermined area including the image display area. Display control method.

Images