Display control device, information terminal, display control method, and display control system

Abstract

This display control device comprises: an image acquisition unit that acquires an image of display data displayed on a monitor mounted on a work machine; a processing unit that identifies the meaning indicated by the display data; and a display control unit that causes a display device to display the identified meaning.

Description

Display control device, information terminal, display control method, and display control system

[0001] The present disclosure relates to a display control device, an information terminal, a display control method, and a display control system.

[0002] 2. Description of the Related Art In the technical field of work machines, a work machine equipped with a monitor, such as that disclosed in Patent Document 1, is known.

[0003] International Publication No. 2016 / 013686

[0004] There is a demand for technology that allows the operator of a work machine to easily understand the meaning of the data displayed on the monitor.

[0005] The present disclosure aims to provide an operator with the meaning of the display data displayed on the monitor.

[0006] According to the present disclosure, there is provided a display control device comprising an image acquisition unit that acquires an image of display data displayed on a monitor mounted on a work machine, a processing unit that identifies the meaning indicated by the display data, and a display control unit that displays the meaning on a display device.

[0007] According to the present disclosure, it is possible to provide the operator with the meaning of the display data displayed on the monitor.

[0008] FIG. 1 is a diagram showing a display control system according to an embodiment. FIG. 2 is a diagram showing a cab of a work machine according to an embodiment. FIG. 3 is a block diagram showing an information terminal according to an embodiment. FIG. 4 is a block diagram showing a terminal computer according to an embodiment. FIG. 5 is a diagram showing a monitor according to an embodiment. FIG. 6 is a diagram for explaining a method of using an information terminal according to an embodiment. FIG. 7 is a diagram for explaining a method of using an information terminal according to an embodiment. FIG. 8 is a diagram for explaining the correspondence between symbol marks according to an embodiment and the meanings indicated by the symbol marks. FIG. 9 is a diagram showing a display device of an information terminal according to an embodiment. FIG. 10 is a flowchart showing a display control method for an information terminal according to an embodiment.

[0009] Hereinafter, embodiments of the present disclosure will be described, but the present disclosure is not limited to the embodiments. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0010] [Display Control System] Fig. 1 is a diagram showing a display control system 100 according to an embodiment. Fig. 2 is a diagram showing a cab 2 of a work machine 1 according to an embodiment. The work machine 1 operates at a work site. In the embodiment, the work machine 1 is a hydraulic excavator.

[0011] The display control system 100 includes a work machine 1 and an information terminal 10. The work machine 1 includes a traveling body 3, a revolving body 4, a work implement 5, a hydraulic cylinder 6, an operation device 7, and a monitor 8.

[0012] The running body 3 runs while supporting the rotating body 4. The running body 3 has a pair of tracks 3A. The running body 3 runs due to the rotation of the tracks 3A. The work machine 1 can move around the work site by using the running body 3.

[0013] The rotating body 4 is supported by the running body 3. The rotating body 4 is disposed above the running body 3. The rotating body 4 rotates while being supported by the running body 3. The operator's cab 2 is provided on the rotating body 4.

[0014] The work machine 5 is supported by the rotating body 4. The work machine 5 performs work. In the embodiment, the work performed by the work machine 5 includes an excavation work of excavating an excavation target and a loading work of loading the excavated material onto a loading target.

[0015] The work implement 5 includes a boom 5A, an arm 5B, and a bucket 5C. The base end of the boom 5A is rotatably connected to the front portion of the revolving unit 4. The base end of the arm 5B is rotatably connected to the tip portion of the boom 5A. The base end of the bucket 5C is rotatably connected to the tip portion of the arm 5B.

[0016] The hydraulic cylinders 6 operate the work implement 5. The hydraulic cylinders 6 include a boom cylinder 6A, an arm cylinder 6B, and a bucket cylinder 6C. The boom cylinder 6A raises and lowers the boom 5A. The arm cylinder 6B causes the arm 5B to perform excavation and dumping operations. The bucket cylinder 6C causes the bucket 5C to perform excavation and dumping operations.

[0017] As shown in Fig. 2, the operating device 7 is disposed in the cab 2. The operating device 7 is operated to operate at least one of the traveling body 3, the revolving body 4, and the work machine 5. The operating device 7 is operated by an operator seated in the cab 2. The operator can operate the operating device 7 while seated in an operator's seat 9 disposed in the cab 2.

[0018] The operating device 7 includes a left operating lever 7A and a right operating lever 7B, a left traveling lever 7C and a right traveling lever 7D, and a left foot pedal 7E and a right foot pedal 7F. The left operating lever 7A is operated to rotate the revolving unit 4. The left operating lever 7A is also operated to operate the work implement 5. The right operating lever 7B is also operated to operate the work implement 5. The left traveling lever 7C and the right traveling lever 7D are operated to operate the traveling unit 3. The left foot pedal 7E is interlocked with the left traveling lever 7C. The right foot pedal 7F is interlocked with the right traveling lever 7D. By operating the left foot pedal 7E and the right foot pedal 7F, the traveling unit 3 may be moved forward or backward.

[0019] The monitor 8 is mounted on the work machine 1. The monitor 8 is disposed in the operator's cab 2. The monitor 8 is disposed to the right front of the operator's seat 9. The monitor 8 has a display device 8A and an input device 8B. Examples of the display device 8A include a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD). The input device 8B generates input data when operated by the operator. Examples of the input device 8B include button switches, a computer keyboard, and a touch panel. Note that the display device 8A and the input device 8B may be separate entities.

[0020] The display device 8A of the monitor 8 provides display data to the operator in the cab 2. The monitor 8 displays the status of the work machine 1. Examples of the status of the work machine 1 include the engine coolant temperature, hydraulic oil temperature, fuel amount, and an image of the surroundings of the work machine 1. In the following description, the display data displayed on the display device 8A of the monitor 8 will be appropriately referred to as the display data displayed on the monitor 8.

[0021] 1, the operator carries an information terminal 10. The operator enters the driver's cab 2 while carrying the information terminal 10.

[0022] 3 is a block diagram showing an information terminal 10 according to an embodiment. The information terminal 10 includes a terminal computer 11, an input device 12, a display device 13, a camera 14, and a battery 15.

[0023] The terminal computer 11 includes a computer system. In this embodiment, application software 16 is installed in the terminal computer 11 to provide the operator with the meaning of the display data displayed on the monitor 8.

[0024] The input device 12 generates input data when operated by an operator. An example of the input device 12 is a touch panel (touch sensor). The input device 12 may include at least one of a computer keyboard and a computer mouse. The input device 12 may also include a voice input device.

[0025] The display device 13 provides display data to the operator. The display device 13 has a display screen on which the display data is displayed. Examples of the display device 13 include a flat panel display such as a liquid crystal display or an organic EL display.

[0026] The camera 14 captures an image of the target object, including display data displayed on the monitor 8.

[0027] The battery 15 functions as a power source for the information terminal 10. The terminal computer 11, the input device 12, the display device 13, and the camera 14 each operate using power supplied from the battery 15.

[0028] 4 is a block diagram showing the terminal computer 11 according to the embodiment. The terminal computer 11 of the information terminal 10 has a processor 20 such as a CPU (Central Processing Unit), a main memory 30 including a non-volatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory), a storage 40, and an interface 50 including an input / output circuit.

[0029] The processor 20 includes an image acquisition unit 21, a processing unit 22, and a display control unit 23. The storage 40 includes an application storage unit 41 and a correspondence relationship storage unit 42.

[0030] The application storage unit 41 stores application software 16. The functions of the terminal computer 11 are stored in the application storage unit 41 of the storage 40 as application software 16, which is a computer program. The processor 20 reads the application software 16 from the application storage unit 41, loads it into the main memory 30, and executes processing in accordance with the application software 16. The application software 16 may be distributed to the terminal computer 11 via a network.

[0031] The image acquisition unit 21 acquires an image of the imaging target captured by the camera 14. In the embodiment, the camera 14 captures display data displayed on the monitor 8. The image acquisition unit 21 acquires an image of the display data displayed on the monitor 8. The image acquisition unit 21 acquires the image of the display data captured by the camera 14.

[0032] The processing unit 22 identifies the meaning indicated by the display data acquired by the image acquisition unit 21. The correspondence storage unit 42 stores the correspondence between the display data and the meaning indicated by the display data. The processing unit 22 identifies the meaning indicated by the display data based on the correspondence stored in the correspondence storage unit 42.

[0033] The display control unit 23 causes the display device 13 to display the meaning indicated by the display data identified by the processing unit 22. The correspondence storage unit 42 stores associated data related to the meaning indicated by the display data. The display control unit 23 causes the display device 13 to display the associated data. In the following description, the display data displayed on the display device 13 of the information terminal 10 will be appropriately referred to as the display data displayed on the information terminal 10.

[0034] [Monitor] Figure 5 is a diagram showing the monitor 8 according to the embodiment. The monitor 8 is mounted on the work machine 1. The monitor 8 displays the status of the work machine 1. In the example shown in Figure 5, the monitor 8 displays, as the status of the work machine 1, a first indicator 61 that indicates the engine coolant temperature, a second indicator 62 that indicates the hydraulic oil temperature, and a third indicator 63 that indicates the amount of fuel.

[0035] The monitor 8 displays symbol marks 64 that indicate the status of the work machine 1. There are a plurality of symbol marks 64 that are displayed on the monitor 8. In the example shown in Fig. 5, the symbol marks 64 include a first symbol mark 64A that indicates the engine coolant temperature, a second symbol mark 64B that indicates the hydraulic oil temperature, and a third symbol mark 64C that indicates the amount of fuel.

[0036] The symbol marks 64 displayed on the monitor 8 are not limited to the first symbol mark 64A indicating the engine coolant temperature, the second symbol mark 64B indicating the hydraulic oil temperature, and the third symbol mark 64C indicating the fuel amount, but other symbol marks 64 may be displayed on the monitor 8. The other symbol marks 64 may be displayed on the monitor 8 together with at least one of the first indicator 61 indicating the engine coolant temperature, the second indicator 62 indicating the hydraulic oil temperature, and the third indicator 63 indicating the fuel amount. An example of the other symbol marks 64 is a caution mark. In the example shown in FIG. 5 , the first symbol mark 64A is displayed in the center of the first indicator 61, the second symbol mark 64B is displayed in the center of the second indicator 62, and the third symbol mark 64C is displayed in the center of the third indicator 63, but the symbol marks 64 may be displayed in any area of ​​the monitor 8.

[0037] The symbol mark 64 can indicate the normal or abnormal state of the work machine 1. When the state of the work machine 1 is normal, the symbol mark 64 is displayed in a first color. When the state of the work machine 1 is abnormal, the symbol mark 64 is displayed in a second color. The first color is, for example, blue or green, and the second color is, for example, yellow or red.

[0038] For example, if the engine coolant temperature is normal, the first symbol mark 64A is displayed in a first color. If the engine coolant temperature is abnormal, the first symbol mark 64A is displayed in a second color. If the hydraulic oil temperature is normal, the second symbol mark 64B is displayed in a first color. If the hydraulic oil temperature is abnormal, the second symbol mark 64B is displayed in a second color. If the fuel amount is normal, the third symbol mark 64C is displayed in a first color. If the fuel amount is normal, the third symbol mark 64C is displayed in a second color.

[0039] In this way, even if the symbol marks 64 have the same shape, the meaning of the symbol marks 64 may differ depending on the color. Even if the first symbol marks 64A have the same shape, the first symbol mark 64A in the first color means that the engine coolant temperature is normal, and the first symbol mark 64A in the second color means that the engine coolant temperature is abnormal. The same applies to the second symbol mark 64B and the third symbol mark 64C.

[0040] In the embodiment, the information terminal 10 provides the operator with the meaning indicated by the symbol mark 64. If the operator does not fully understand the meaning indicated by the symbol mark 64, the operator captures an image of the symbol mark 64 with the camera 14 of the information terminal 10.

[0041] 6 and 7 are diagrams for explaining how to use the information terminal 10 according to the embodiment. As shown in Fig. 6, when an operator does not fully understand the meaning of the symbol mark 64, the operator captures an image of the symbol mark 64 displayed on the monitor 8 with the camera 14 of the information terminal 10. As shown in Fig. 7, when the symbol mark 64 is captured by the camera 14, an image of the symbol mark 64 is displayed on the information terminal 10.

[0042] As an example, if the operator does not understand the meaning of the first symbol mark 64A and captures an image of the first symbol mark 64A displayed on the monitor 8 with the camera 14 of the information terminal 10, an image of the first symbol mark 64A will be displayed on the information terminal 10, as shown in Figure 7.

[0043] The image acquisition unit 21 acquires an image of the first symbol mark 64A captured by the camera 14. The processing unit 22 identifies the first symbol mark 64A. The processing unit 22 also identifies the meaning indicated by the first symbol mark 64A. The correspondence storage unit 42 stores the correspondence between the symbol marks 64 and the meanings indicated by the symbol marks 64. As described above, even symbol marks 64 of the same shape may have different meanings depending on their colors. The correspondence storage unit 42 may store the relationship between the symbol marks 64 and the meanings indicated by the symbol marks 64 for each color. The processing unit 22 identifies the first symbol mark 64A and the meaning indicated by the first symbol mark 64A based on the correspondence stored in the correspondence storage unit 42.

[0044] Figure 8 is a diagram for explaining the correspondence between symbol marks 64 and meanings 65 indicated by the symbol marks 64 according to this embodiment. As shown in Figure 8, there are a plurality of symbol marks 64 displayed on the monitor 8. A meaning 65 is assigned to each of the plurality of symbol marks 64. The meaning 65 indicated by the symbol marks 64 includes the state of the work machine 1. The state of the work machine 1 includes the working state of the work machine 1. The state of the work machine 1 includes the state of the equipment of the work machine 1. The state of the work machine 1 includes the state of consumables of the work machine 1. A correspondence table indicating the correspondence between the symbol marks 64 and the meanings 65 indicated by the symbol marks 64 is stored in advance in the correspondence storage unit 42.

[0045] The processing unit 22 identifies the symbol mark 64 acquired by the image acquisition unit 21. The processing unit 22 may identify the symbol mark 64 acquired by the image acquisition unit 21 based on a template matching method. The processing unit 22 may identify the symbol mark 64 acquired by the image acquisition unit 21 based on an artificial intelligence (AI) algorithm. The AI ​​algorithm identifies the symbol mark 64 acquired by the image acquisition unit 21 based on a learning model generated by learning feature quantities of the symbol mark 64. The learning model receives the feature quantities of the symbol mark 64 as input and outputs the type of the symbol mark 64. The processing unit 22 can identify the type of the symbol mark 64 acquired by the image acquisition unit 21 by inputting the image of the symbol mark 64 acquired by the image acquisition unit 21 into the learning model. The processing unit 22 can identify the symbol mark 64 acquired by the image acquisition unit 21 as, for example, the first symbol mark 64A by inputting the image of the symbol mark 64 acquired by the image acquisition unit 21 into the learning model.

[0046] After identifying the type of the symbol mark 64 acquired by the image acquisition unit 21, the processing unit 22 compares the symbol mark 64 with the correspondence table shown in Fig. 8 to identify the meaning 65 indicated by the symbol mark 64. For example, if the processing unit 22 identifies that the symbol mark 64 acquired by the image acquisition unit 21 is the first symbol mark 64A, the processing unit 22 can identify the meaning 65 indicated by the first symbol mark 64A based on the correspondence table stored in the correspondence storage unit 42.

[0047] The display control unit 23 causes the information terminal 10 to display the meaning 65 indicated by the first symbol mark 64A identified by the processing unit 22.

[0048] In the embodiment, the plurality of symbol marks 64 displayed on the monitor 8 may be captured by the camera 14 of the information terminal 10. In this case, the display control unit 23 may simultaneously display, on the information terminal 10, the plurality of meanings indicated by the plurality of symbol marks 64 identified by the processing unit 22, or may sequentially display, on the information terminal 10, the plurality of meanings indicated by the plurality of symbol marks 64 identified by the processing unit 22 one by one.

[0049] 9 is a diagram illustrating the display device 13 of the information terminal 10 according to the embodiment. For example, if the symbol mark 64 acquired by the image acquisition unit 21 is a first symbol mark 64A, the display control unit 23 displays an icon of the first symbol mark 64A and a meaning 65 (short description) indicated by the first symbol mark 64A on the display device 13, as shown in FIG. 9. By checking the information terminal 10, the operator can understand that the meaning 65 indicated by the first symbol mark 64A is the engine coolant temperature. If the meaning 65 indicated by the first symbol mark 64A is an abnormality in the engine coolant temperature, the operator can understand that the meaning 65 indicated by the first symbol mark 64A is an abnormality in the engine coolant temperature by checking the information terminal 10.

[0050] In the embodiment, the display control unit 23 causes the display device 13 to display associated data related to the meaning 65 indicated by the symbol mark 64. The correspondence storage unit 42 stores associated data related to the meaning 65 indicated by the symbol mark 64. The display control unit 23 causes the display device 13 to display associated data related to the meaning 65 indicated by the first symbol mark 64A, based on the meaning 65 indicated by the symbol mark 64 identified by the processing unit 22 and the associated data stored in the correspondence storage unit 42.

[0051] The associated data includes an explanatory document relating to the state of the work machine 1. The correspondence storage unit 42 stores the instruction manual for the work machine 1. If the symbol mark 64 identified by the processing unit 22 is the first symbol mark 64A, the associated data includes an explanatory document relating to the engine coolant temperature. The processing unit 22 extracts the explanatory document relating to the engine coolant temperature from the instruction manual. The display control unit 23 displays the explanatory document extracted by the processing unit 22 on the display device 13. As shown in FIG. 9 , in an embodiment, the display control unit 23 displays a summarized document and the full explanation of the instruction manual on the display device 13. The operator can easily check the instruction manual.

[0052] As described above, the meaning 65 indicated by the symbol mark 64 includes normal and abnormal conditions of the work machine 1. If the meaning 65 indicated by the symbol mark 64 identified by the processing unit 22 is an abnormal condition of the work machine 1, the display control unit 23 causes the display device 13 to display, as related data, a method of dealing with the abnormal condition of the work machine 1. For example, if the meaning 65 indicated by the first symbol mark 64A is an abnormality in the engine coolant temperature, the display control unit 23 causes the display device 13 to display text recommending warm-up operation as a method of dealing with the abnormality in the coolant temperature. The operator can easily check the method of dealing with the abnormality.

[0053] 10 is a flowchart showing a display control method for the information terminal 10 according to the embodiment. For example, when an operator does not understand the meaning of the first symbol mark 64A displayed on the monitor 8, the operator captures an image of the first symbol mark 64A displayed on the monitor 8 with the camera 14 of the information terminal 10. The image acquisition unit 21 acquires an image of the first symbol mark 64A (step S1).

[0054] The processing unit 22 identifies the type of the symbol mark 64 acquired by the image acquisition unit 21 and the meaning 65 indicated by the symbol mark 64 based on the correspondence table stored in the correspondence storage unit 42 (step S2).

[0055] After identifying the symbol mark 64 as the first symbol mark 64A, the processing unit 22 obtains associated data related to the meaning 65 indicated by the first symbol mark 64A from the correspondence storage unit 42. The processing unit 22 searches, for example, from instruction manuals stored in the correspondence storage unit 42, for an explanatory document related to the engine coolant temperature indicated by the first symbol mark 64A (step S3).

[0056] The display control unit 23 causes the display device 13 to display the meaning 65 indicated by the first symbol mark 64A. As described with reference to Fig. 9, the display control unit 23 also causes the display device 13 to display the explanatory document and the solution as related data (step S4).

[0057] [Effects] As described above, in the embodiment, the terminal computer 11 functioning as a display control device includes an image acquisition unit 21 that acquires an image of the symbol mark 64, which is at least a part of the display data displayed on the monitor 8 mounted on the work machine 1, a processing unit 22 that identifies the meaning 65 indicated by the symbol mark 64, and a display control unit 23 that causes the meaning 65 indicated by the symbol mark 64 to be displayed on the display device 13.

[0058] According to the embodiment, when the operator cannot fully understand the meaning 65 indicated by the symbol mark 64 displayed on the monitor 8, the meaning 65 indicated by the symbol mark 64 is provided to the operator from the information terminal 10. This allows the operator to understand the meaning indicated by the symbol mark 64.

[0059] A correspondence table showing the correspondence between the symbol marks 64 and the meanings 65 indicated by the symbol marks 64 is stored in advance in the correspondence storage unit 42. The processing unit 22 can identify the meanings 65 indicated by the symbol marks 64 captured by the camera 14 by comparing the image of the symbol marks 64 acquired by the image acquisition unit 21 with the correspondence table.

[0060] The display control unit 23 causes the display device 13 to display an explanatory document and a method of dealing with the problem as related data relating to the meaning 65 indicated by the symbol mark 64. This allows the operator to smoothly deal with any abnormalities in the condition of the work machine 1.

[0061] [Other Embodiments] In the above-described embodiment, the terminal computer 11 identifies the meaning indicated by the symbol mark 64 displayed on the monitor 8. The terminal computer 11 may also identify the meaning indicated by at least one of a character string, a numeric string, and a symbol displayed on the monitor 8.

[0062] 1...working machine, 2...operator's cab, 3...traveling body, 3A...track, 4...swivel body, 5...working machine, 5A...boom, 5B...arm, 5C...bucket, 6...hydraulic cylinder, 6A...boom cylinder, 6B...arm cylinder, 6C...bucket cylinder, 7...operating device, 7A...left operating lever, 7B...right operating lever, 7C...left traveling lever, 7D...right traveling lever, 7E...left foot pedal, 7F...right foot pedal, 8...monitor, 8A...display device, 8B...input device, 9...operator's seat, 10...information terminal, 11...terminal computer, 12...input device, 13... Display device, 14...camera, 15...battery, 16...application software, 20...processor, 21...image acquisition unit, 22...processing unit, 23...display control unit, 30...main memory, 40...storage, 41...application storage unit, 42...correspondence storage unit, 50...interface, 61...first indicator, 62...second indicator, 63...third indicator, 64...symbol mark, 64A...first symbol mark, 64B...second symbol mark, 64C...third symbol mark, 65...meaning, 100...display control system.

Claims

1. A display control device comprising: an image acquisition unit that acquires an image of display data displayed on a monitor mounted on a work machine; a processing unit that identifies the meaning indicated by the display data; and a display control unit that displays the meaning on a display device.

2. The display control device according to claim 1, wherein the image acquisition unit acquires an image of the display data captured by a camera.

3. The display control device according to claim 1, further comprising a correspondence relationship storage unit that stores a correspondence relationship between the display data and the meaning, and the processing unit identifies the meaning based on the correspondence relationship.

4. The display control device according to claim 3, wherein the correspondence storage unit stores associated data related to the meaning, and the display control unit causes the display device to display the associated data.

5. A display control device according to claim 4, wherein the meaning includes a status of the work machine, and the related data includes an explanatory document relating to the status.

6. A display control device according to claim 4, wherein the meaning includes an abnormality in the condition of the work machine, and the related data includes a method of dealing with the abnormality.

7. The display control device according to claim 1, wherein the display data includes a symbol mark indicating the status of the work machine.

8. An information terminal comprising the display control device according to claim 1.

9. An information terminal according to claim 8, further comprising a camera, wherein the image acquisition unit of the display control device acquires an image of the display data captured by the camera.

10. A display control method, in which a computer executes the following steps: acquiring an image of display data displayed on a monitor mounted on a work machine; identifying the meaning indicated by the display data; and displaying the meaning on a display device.

11. A display control system comprising: an image acquisition unit that acquires an image of display data displayed on a monitor mounted on a work machine; a processing unit that identifies the meaning indicated by the display data; and a display control unit that displays the meaning on a display device.

Images