Video display system

Abstract

The present invention provides a video display system for displaying video of work status that can easily suppress the decline in visibility of the work status and enable the work to be performed appropriately. [Solution] The video display system 100 comprises a traveling device 11, a working device 12 for working on a specific part 12a, an imaging device 13, and a display device 21, and includes a position acquisition unit 13a1 for acquiring the position of the specific part 12a, a parameter determination unit 13a2 for determining the imaging direction and magnification of the imaging device 13 as parameters that define the video based on at least the acquired position of the specific part 12a, and an imaging instruction unit 13a3 for instructing the imaging device 13 to capture a video with the determined imaging direction and magnification.

Description

[Technical Field]

[0001] This invention relates to a system for displaying video footage of the working conditions of work machines and the like. [Background technology]

[0002] Conventionally, technologies for displaying images of the working conditions of work machines and the like are known. For example, Patent Documents 1 and 2 disclose a system comprising an imaging device and a display device that displays images of the working conditions captured by the imaging device. In this system, the display device is located away from the work machine, and the captured images are displayed remotely. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2018-121195 [Patent Document 2] Japanese Patent Publication No. 2023-226094 [Overview of the Initiative]

[0004] According to the technologies described in Patent Documents 1 and 2 above, captured images can be viewed remotely. However, the field of view of the captured images is constant, and if a specific part of the work device performing the work (for example, the tip) is displaced, the visibility of the work situation may decrease. This makes it difficult to perform the work properly. In addition, to suppress the decrease in visibility, it is conceivable to place multiple imaging devices at different positions to capture the work situation. In this case, multiple imaging devices are required, and the system becomes more complex, so improvement is desired.

[0005] In view of the above, the object of the present invention is to provide a system for displaying video of work conditions that can easily suppress a decrease in the visibility of the work conditions and enable the work to be performed appropriately. [Means for solving the problem]

[0006] The technical means of the present invention for solving this technical problem are characterized by the following points. The video display system of the present invention is a system for displaying video of work conditions. The video display system of the present invention comprises a traveling device, a work device provided on the traveling device and capable of working at a specific part that is movable relative to the traveling device, an imaging device provided on the traveling device and capturing video of the work condition at the specific part on the work device, and a display device for displaying the video captured by the imaging device. The video display system of the present invention comprises a position acquisition unit for acquiring the position of the specific part on the work device, a parameter determination unit for determining the direction and magnification of imaging in the imaging device as parameters defining the video, based on at least the acquired position of the specific part, and an imaging instruction unit for instructing the imaging device to capture the video with the determined direction and magnification of imaging.

[0007] In the video display system of the present invention, the work device has one end and the other end defined, is connected to the traveling device at the one end, and is configured such that the specific part is located at the other end, and is equipped with a sensor between the one end and the other end that transmits displacement information of the work device, and the position acquisition unit acquires the position of the specific part based on the transmitted displacement information.

[0008] In the video display system of the present invention, the parameter determination unit determines the direction and magnification of imaging in the imaging device based on the position of the specific part as well as the position of the imaging device in the traveling device.

[0009] In the image display system of the present invention, when the position of the specific part is displaced from a predetermined first position to a second position different from the first position, the parameter determination unit calculates the angle between the direction from the position of the imaging device toward the first position and the direction from the position of the imaging device toward the second position, and uses the calculated angle to determine the direction of imaging.

[0010] In the video display system of the present invention, when the position of the specific part is displaced from a predetermined first position to a second position different from the first position, the parameter determination unit calculates the ratio between the distance from the position of the imaging device to the first position and the distance from the position of the imaging device to the second position, and determines the magnification of the imaging using the calculated ratio.

[0011] In the video display system of the present invention, the imaging instruction unit instructs the imaging device to image the video including the image of the specific part in the working device.

Advantages of the Invention

[0012] According to the present invention, in a system for displaying a video of a working situation, it is possible to easily suppress a decrease in the visibility of the working situation and appropriately perform the work.

Brief Description of the Drawings

[0013] [Figure 1] It is an overall schematic diagram of a video display system according to an embodiment of the present invention. [Figure 2] It is a functional block diagram showing parts constituting the working machine shown in FIG. 1. [Figure 3] It is a diagram showing a state where a specific part in the working device of the working machine shown in FIG. 1 is in the first position. <0000°91> [Figure 4] It is a diagram showing a state where a video of the working situation when the specific part is in the first position is displayed on the display device of the terminal shown in FIG. 1. [Figure 5] It is a flowchart showing a series of operations of the video display system shown in FIG. 1. [[ID=3))<C000095> [Figure 6] It is a diagram showing a state where a specific part in the working device of the working machine shown in FIG. 1 is displaced from the first position to the second position. [Figure 7] It is a diagram showing a state where a video of the working situation when the specific part is in the second position is displayed on the display device of the terminal shown in FIG. 1. <0000 <100>This figure shows the state in which a specific part of the working device of the work machine shown in Figure 1 has been displaced from the first position to the second position. [Figure 9] Figure 1 shows the display device of the terminal, displaying video of the work situation when a specific part is in the second position. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below with reference to the drawings.

[0015] As shown in Figure 1, the video display system 100 according to an embodiment of the present invention is applied to a work machine 10. The video display system 100 is a system for displaying video of the work status by the work machine 10. The video display system 100 also includes a terminal 20. The terminal 20 may be installed, for example, at a location separated from the work machine 10. The work machine 10 and the terminal 20 are each capable of communicating information via an information communication network N.

[0016] The video display system 100 comprises a traveling device 11, a working device 12, an imaging device 13, an imaging control device 13a, and a display device 21. The traveling device 11 and the working device 12 constitute the working machine 10. The working device 12 is mounted on the traveling device 11. The working device 12 includes a specific part 12a. The specific device 12a is relative to the traveling device 11 and is configured to perform a predetermined task.

[0017] The work machine 10 is construction machinery, heavy machinery, etc., such as a drill jumbo, breaker, sprayer, backhoe, wheel loader, etc. The work performed by the work device 12 corresponds to specific examples of the work machine 10 described above, such as drilling, marking, spraying, excavation, and mowing. The specific part 12a corresponds to specific examples of the work machine 10 described above, such as the boom tip, chisel tip, nozzle, bucket, crusher input port, etc.

[0018] In this embodiment, as an example, the work machine 10 is described as a drill jumbo. The traveling device 11 is a vehicle that travels on four wheels, and its length, width, and height are defined. In this embodiment, as indicated by the arrows shown in the figures as appropriate, the front-rear direction, left-right direction, and up-down direction correspond to the vehicle length direction, vehicle width direction, and vehicle height direction, respectively. The traveling device 11 may be driven by automatic driving control or remote control.

[0019] The working device 12 is configured in an elongated shape, with one end 121 and the other end 122 defined. One end 121 of the working device 12 is connected to the front of the traveling device 11. A specific part 12a is located on the other end 122 side. Therefore, in this embodiment, the specific part 12a is located at the front end, ahead of the traveling device 11. The specific part 12a is three-dimensionally movable relative to the traveling device 11. The working device 12 may be driven by automatic driving control or remote control.

[0020] The imaging device 13 is mounted on the traveling device 11. More specifically, the imaging device 13 may be installed, for example, behind the connection point (other end 122) of the working device 12 and above the traveling device 11. The imaging device 13 captures images of the working conditions of a specific part 12a of the working device 12. The imaging device 13 allows adjustment of the imaging direction and imaging magnification. The imaging direction can be adjusted, for example, to face any direction in three dimensions.

[0021] The imaging control device 13a performs control such as acquiring the position of a specific part 12a, determining parameters such as the direction and magnification of imaging by the imaging device 13, and issuing imaging instructions to the imaging device 13. In this embodiment, the imaging control device 13a is provided on the traveling device 11 of the work machine 10, but it may be provided on, for example, the imaging device 13, the terminal 20, etc.

[0022] In this embodiment, the display device 21 is installed at a terminal 20 located away from the work machine 10. The display device 21 displays images captured by the imaging device 13. In other words, the display device 21 can remotely display the working status of the work device 12 on the work machine 10, even when it is away from the work machine 10.

[0023] Figure 2 is a functional block diagram showing the various components of the work machine 10. The work machine 10 is equipped with a travel device 11, a work device 12, an imaging device 13, a sensor 14, a communication device 15, a drive control device 11a, and an imaging control device 13a. These are connected to each other so that they can communicate information via an in-vehicle communication network N1 such as CAN.

[0024] Sensor 14 is provided on the work device 12. In this embodiment, sensor 14 is provided between one end 121 and the other end 122 and transmits displacement information of the work device 12. Sensor 14 may be composed of, for example, a gyroscope, an accelerometer, etc. Sensor 14 may, for example, detect the tilt angle in a three-dimensional system as displacement information of the work device 12. The displacement information detected by sensor 14 is sent to the drive control device 11a and the imaging control device 13a.

[0025] The communication device 15 is installed on the traveling device 11. The communication device 15 may, for example, be capable of wireless communication according to a predetermined standard. The communication device 15 transmits the video captured by the imaging device 13 to the terminal 20 via the information communication network N. The communication device 15 also receives information such as control instructions from the terminal 20 via the information communication network N.

[0026] The drive control device 11a is installed on the traveling device 11. The drive control device 11a performs control such as issuing drive instructions to the traveling device 11 and the work device 12. The drive control device 11a consists of a CPU, electrical circuits, etc., and the above-mentioned control is possible through predetermined program processing.

[0027] The imaging control device 13a is provided on the traveling device 11. The imaging control device 13a performs control such as issuing imaging instructions to the imaging device 13. The imaging control device 13a consists of a CPU, electrical circuits, etc., and the above-mentioned control is possible through predetermined program processing. The imaging control device 13a includes a position acquisition unit 13a1, a parameter determination unit 13a2, and an imaging instruction unit 13a3.

[0028] The position acquisition unit 13a1 acquires the position of a specific part 12a on the work device 12. The position acquisition unit 13a1 acquires the position of the specific part 12a based at least on displacement information transmitted by the sensor 14. More specifically, for example, if the displacement information from the sensor 14 is an inclination angle, the position acquisition unit 12a may calculate the three-dimensional coordinates of the specific part 12a based on the inclination angle, the dimensions of each part of the work machine 10, the position information of the work machine 10, etc.

[0029] The parameter determination unit 13a2 determines the imaging direction and magnification of the imaging device 13 as parameters defining the image, based at least on the position of the specific part 12a acquired by the position acquisition unit 13a1. In this embodiment, the parameter determination unit 13a2 determines the imaging direction and magnification of the imaging device 13 based not only on the position of the specific part 12a but also on the position of the imaging device 13 on the traveling device 11. The position of the imaging device 13 may be a pre-measured 3D coordinate system, or the 3D coordinate system of the imaging device 13 may be calculated by the parameter determination unit 13a2 based on the dimensions of each part of the work machine 10, position information of the work machine 10, etc. Note that the position of another part of the work machine 10 may be substituted for the position of the imaging device 13.

[0030] More specifically, in this embodiment, the imaging direction and imaging magnification are determined by the parameter determination unit 13a2 as follows. When the position of a specific part 12a is displaced from a predetermined first position to a second position different from the first position, the parameter determination unit 13a2 calculates the angle formed by the direction from the position of the imaging device 13 toward the first position and the direction from the position of the imaging device 13 toward the second position, and uses the calculated angle to determine the imaging direction.

[0031] The parameter determination unit 13a2 calculates the ratio of the distance from the position of the imaging device 13 to the first position and the distance from the position of the imaging device 13 to the second position when the position of the specific part 12a is displaced from a predetermined first position to a second position different from the first position, and uses the calculated ratio to determine the magnification for imaging.

[0032] As shown in Figure 3, the boom of the work device 12 is assumed to extend approximately horizontally (in the front-rear direction). For example, the position of the specific part 12a in this state may be called the "first position". The position of the specific part 12a and the position of the imaging device 13 may each be in a three-dimensional coordinate system. For example, with the specific part 12a in the first position, the position of the specific part 12a and the position of the imaging device 13 may each be in coordinate (X b0 ,Y b0 ,Z b0 ), and the coordinates (Xc, Yc, Zc) shall be used. In this case, the coordinates (Xc, Yc, Zc) shall be used to represent the coordinates (X b0 ,Y b0 ,Z b0 A vector pointing towards ) is defined.

[0033] On the other hand, if a specific part 12a is displaced from the state shown in Figure 4, using the first position as a reference, the position of the specific part 12a after the displacement may be called the "second position" (see Figures 6 and 8). Even when the specific part 12a is in the second position, the vector is defined in the same way as described above. Therefore, vectors corresponding to the first position and the second position can be obtained. From the absolute value and dot product of each vector, the "angle" described above can be calculated, and from the ratio of the absolute values ​​of each vector, the "ratio" of the distance described above can be calculated.

[0034] As shown in Figure 2, the imaging instruction unit 13a3 instructs the imaging device 13 to capture an image with the imaging direction and magnification determined by the parameter determination unit 13a2. More specifically, the imaging instruction unit 13a3 may instruct the imaging device 13 to capture an image that includes an image of a specific part 12a on the work device 12. That is, the image of the specific part 12a may be included in the image so that the specific part 12a is also displayed within the field of view displayed on the display device 21.

[0035] In the state shown in Figure 3, when the specific part 12a is in the first position, the imaging device 13 captures an image of the work situation. At this time, it is preferable to adjust the imaging direction and magnification in advance so that the image of the specific part 12a is included in the field of view. In this case, as shown in Figure 4, the display device 21 displays the work situation near the work device 12 within the field of view, as well as the specific part 12a. The specific part 12a within the field of view may be displayed at a position slightly off-center, or it may be displayed approximately in the center.

[0036] Figure 5 is a flowchart showing a series of operations in the video display system 100. When imaging the working status of the work machine 10 and displaying the image, first, in step S1, the imaging control device 13a determines whether or not a specific part 12a has been displaced from a first position to a second position. This determination may also be made based on the detection value of the sensor 14, and if it is determined that the tilt angle of the work machine 12 has changed, it may be determined that "a specific part 12a has been displaced from a first position to a second position."

[0037] At present, it is assumed that the specific part 12a is in the first position and the state in FIG. 3 is maintained. Note that the vector from the coordinates (Xc, Yc, Zc) to the coordinates (X b0 , Y b0 , Z b0 ) is separately calculated by other arithmetic processes and stored in a memory or the like. In this case, it is determined as "No" in step S1, and the process proceeds to step S2.

[0038] In step S2, the imaging control device 13a (imaging instruction unit 13a3) instructs the imaging device 13 to perform imaging. Accordingly, the imaging device 13 captures an image. The imaging direction and magnification have been adjusted in advance, and the captured image includes an image of the specific part 12a within the viewing angle.

[0039] Next, in step S3, the image captured in step S2 is transmitted from the work machine 10 (communication device 15) to the terminal 20 via the information communication network N. Next, in step S4, the image transmitted in step S3 is displayed on the display device 21 (see FIG. 4).

[0040] On the other hand, a case where the specific part 12a that was in the first position is displaced from the first position to the second position will be described. For example, as shown in FIG. 6, it is assumed that the work device 12 that was substantially horizontal is inclined and the specific part 12a is located in the front upper oblique side. The position of the specific part 12a in this state is the second position. In this case, it is determined as "Yes" in step S1 of FIG. 5, and the process proceeds to step S5.

[0041] In step S5, the imaging control device 13a (position acquisition unit 13a1) acquires the position of the specific part 12a. In this case, as shown in FIG. 6, the coordinates (X b1 , Y b1 , Z b1 ) are acquired as the position of the specific part 12a.

[0042] Next, in step S6 of Figure 5, the imaging control device 13a (parameter determination unit 13a2) determines and updates the imaging direction and magnification. As shown in Figure 6, the coordinates (Xc, Yc, Zc) are changed to coordinates (X b1 ,Y b1 ,Z b1 A vector pointing to (Xc, Yc, Zc) and a vector pointing to coordinate (X b0 ,Y b0 ,Z b0 Based on the vectors pointing to ) and , the angle and distance ratio between these vectors are calculated. The imaging direction and magnification are updated by correction using the calculated angle and distance ratio.

[0043] Next, in step S2 of Figure 5, the imaging control device 13a (imaging instruction unit 13a3) instructs the imaging device 13 to take an image. In response, the imaging device 13 takes an image. The imaging direction and magnification are determined and updated in step S6, and the image taken will include the image of the specific part 12a in its field of view.

[0044] After the subsequent steps S3 and S4, as shown in Figure 7, the display device 21 displays the work status near the work device 12 within its field of view, as well as the specific part 12a.

[0045] Furthermore, for example, as shown in Figure 8, suppose the working device 12, which was previously nearly horizontal, becomes tilted so that a specific part 12a is positioned diagonally downward and forward. In this state, the position of the specific part 12a is the second position. In this case, "Yes" is determined in step S1 of Figure 5, and the process proceeds to step S5.

[0046] In step S5, the imaging control device 13a (position acquisition unit 13a1) acquires the position of the specific part 12a. In this case, as shown in Figure 8, the position of the specific part 12a is given by coordinate (X b2 ,Y b2 ,Z b2 ) is obtained.

[0047] Next, in step S6 of Figure 5, the imaging control device 13a (parameter determination unit 13a2) determines and updates the imaging direction and magnification. As shown in Figure 8, the coordinates (Xc, Yc, Zc) are changed to coordinates (X b2 ,Y b2 ,Z b2 A vector pointing to (Xc, Yc, Zc) and a vector pointing to coordinate (X b0 ,Y b0 ,Z b0 Based on the vectors pointing to ) and , the angle and distance ratio between these vectors are calculated. The imaging direction and magnification are updated by correction using the calculated angle and distance ratio.

[0048] Next, in step S2 of Figure 5, the imaging control device 13a (imaging instruction unit 13a3) instructs the imaging device 13 to take an image. In response, the imaging device 13 takes an image. The imaging direction and magnification are determined and updated in step S6, and the image taken will include the image of the specific part 12a in its field of view.

[0049] Following steps S3 and S4, as shown in Figure 9, the display device 21 displays the work status near the work device 12 within its field of view, as well as the specific part 12a.

[0050] [Effects of the Embodiment] As described above, the video display system 100 according to the embodiment of the present invention is a system for displaying video of work status. The video display system 100 comprises a traveling device 11, a work device 12 provided on the traveling device 11 and performing work at a specific part 12a that is movable relative to the traveling device 11, an imaging device 13 provided on the traveling device 11 and capturing video of the work status of the specific part 12a on the work device 12, and a display device 21 that displays the video captured by the imaging device 13. The video display system 100 comprises a position acquisition unit 13a1 that acquires the position of the specific part 12a on the work device 12, a parameter determination unit 13a2 that determines the direction and magnification of imaging in the imaging device 13 as parameters defining the video based on the acquired position of the specific part 12a, and an imaging instruction unit 13a3 that instructs the imaging device 13 to capture the video with the determined imaging direction and magnification.

[0051] According to this, even if the position of a specific part 12a is displaced, the imaging direction and magnification are appropriately adjusted to capture an image of the work situation. Therefore, in a system for displaying an image of the work situation, a decrease in the visibility of the work situation can be easily suppressed, and the work can be performed appropriately.

[0052] The work device 12 has one end 121 and the other end 122 defined, and is connected to the traveling device 11 at the one end 121, with the specific part 12a located at the other end 122. A sensor 14 is provided between the one end 121 and the other end 122 to transmit displacement information of the work device 12, and the position acquisition unit 13a1 acquires the position of the specific part 12a based at least on the transmitted displacement information.

[0053] According to this, the position of a specific part 12a can be obtained accurately and easily based on the displacement information from the sensor 14.

[0054] The parameter determination unit 13a2 determines the imaging direction and magnification of the imaging device 13 based on the position of the specific part 12a as well as the position of the imaging device 13 on the traveling device 11.

[0055] According to this, the direction and magnification of imaging can be easily determined using the position of the specific part 12a and the position of the imaging device 13.

[0056] The parameter determination unit 13a2 calculates the angle between the direction from the position of the imaging device 13 toward the first position and the direction from the position of the imaging device 13 toward the second position when the position of the specific part 12a is displaced from a predetermined first position to a second position different from the first position, and uses the calculated angle to determine the direction of imaging.

[0057] According to this method, even if the position of a specific part 12a is displaced from the first position to the second position, the direction of imaging can be easily and accurately determined using two vectors or the like.

[0058] The parameter determination unit 13a2 calculates the ratio of the distance from the position of the imaging device 13 toward the first position to the distance from the position of the imaging device 13 toward the second position when the position of the specific part 12a is displaced from a predetermined first position to a second position different from the first position, and uses the calculated ratio to determine the magnification of the imaging.

[0059] According to this method, even if the position of a specific part 12a is displaced from the first position to the second position, the imaging magnification can be easily and accurately determined using two vectors or the like.

[0060] The imaging instruction unit 13a3 instructs the imaging device 13 to capture the image that includes the image of the specific part 12a of the work device 12.

[0061] According to this, even if the position of the specific part 12a is displaced from the first position to the second position, the image of the specific part 12a can be included in the image displayed on the display device 21. Therefore, the work can be performed more appropriately. [Explanation of Symbols]

[0062] 10...Working machine, 11...Traveling device, 12...Working device, 12a...Specific part, 13...Imaging device, 13a...Imaging control device, 13a1...Position acquisition unit, 13a2...Parameter determination unit, 13a3...Imaging instruction unit, 14...Sensor, 20...Terminal, 21...Display device, 100...Video display system

Claims

1. Traveling device and A work device provided on the aforementioned traveling device, which performs work at a specific part that is movable relative to the traveling device, An imaging device provided on the aforementioned traveling device for capturing images of the work status by the specific part of the work device, A display device that displays the image captured by the aforementioned imaging device, Equipped with In a video display system for displaying video of the aforementioned work status, A position acquisition unit that acquires the position of the specific part in the work device, At a minimum, a parameter determination unit that determines the direction and magnification of imaging in the imaging device as parameters defining the image based on the acquired position of the specific part, An imaging instruction unit that instructs the imaging device to capture the image with the determined imaging direction and magnification, Equipped with Video display system.

2. In the video display system according to claim 1, The aforementioned work apparatus is One end and the other end are defined, and the one end is connected to the traveling device, and the other end is configured to be located at the specific part, and a sensor is provided between the one end and the other end to transmit displacement information of the working device, The position acquisition unit, At a minimum, the position of the specific part is obtained based on the displacement information transmitted. Video display system.

3. In the video display system according to claim 2, The parameter determination unit, Based on the position of the specific part mentioned above, as well as the position of the imaging device on the traveling device, the direction and magnification of imaging by the imaging device are determined. Video display system.

4. In the video display system according to claim 3, The parameter determination unit, When the position of the specified part is displaced from a predetermined first position to a second position different from the first position, the angle between the direction from the position of the imaging device toward the first position and the direction from the position of the imaging device toward the second position is calculated, and the direction of imaging is determined using the calculated angle. Video display system.

5. In the video display system according to claim 3 or claim 4, The parameter determination unit, When the position of the specific part is displaced from a predetermined first position to a second position different from the first position, the ratio of the distance from the position of the imaging device toward the first position to the distance from the position of the imaging device toward the second position is calculated, and the magnification of the imaging is determined using the calculated ratio. Video display system.

6. In the video display system according to any one of claims 1 to 4, The imaging instruction unit is, The imaging device is instructed to capture the image that includes the image of the specific part of the work apparatus. Video display system.

7. In the video display system according to claim 5, The imaging instruction unit is, The imaging device is instructed to capture the image that includes the image of the specific part of the work apparatus. Video display system.

Images