Apparatus, operability estimating method, and program

Abstract

This apparatus makes it possible to estimate operability with regard to remote operation of a work machine, taking into account communication delay and video quality, by including: a delay operability estimating unit configured to calculate a delay operability, which is an estimated value of operability based on a delay, on the basis of the characteristic that the operability of the work machine decreases as a delay in communication relating to operation during remote operation of the work machine increases, but the operability does not decrease if the delay is less than a predetermined time; a video operability estimating unit configured to calculate a video operability, which is an estimated value of operability based on the quality of a video, on the basis of the characteristic that, in relation to communication of a video of the work machine during remote operation, operability is high until a certain bit rate, and operability decreases as the packet loss rate increases; and an integrating unit configured to calculate an estimated value of the operability of the work machine on the basis of the delay operability and the video operability.

Description

Apparatus, operability estimation method, and program

[0001] The present invention relates to a device, an operability estimation method, and a program.

[0002] Remote control systems that enable heavy machinery and other construction equipment in remote locations to be operated from a control room have been developed to reduce the risk of accidents during on-site work and to alleviate labor shortages. With these remote control systems, it is necessary to consider the risk of accidents due to reduced operability caused by a deterioration in network quality during work.

[0003] This section explains the impact of a decrease in network quality during remote operation on operability. In remote operation, operation information from an operating device in a control room is transmitted to a remote work machine, causing the work machine to operate, and the operation of the work machine is transmitted to the control room as a camera-captured image. Therefore, a decrease in the quality (communication quality) of the network transmitting the operation information and image is likely to degrade the operator's operability. For example, an increase in packet loss rate due to a decrease in network quality or a decrease in image quality due to a decrease in encoding bit rate associated with a decrease in throughput can make it difficult to visually grasp the situation in a remote location, thereby degrading operability. Furthermore, increased network latency increases the reaction time of the work machine in the image to the operator's operation, creating a discrepancy between the operator's operation and the work machine's operation in the image, degrading operability (Non-Patent Document 1).

[0004] Considering the above-mentioned effects, it is important to quickly grasp changes in video quality and latency, as well as degradation of operability due to a combination of these factors. Therefore, a technology is needed to estimate operability by acquiring information on video quality and latency that cause degradation of operability from information on the devices and network that make up the remote control and grasping the impact of this information on operability. As a similar method for estimating operability, ITU-T Recommendation G.1072 (Non-Patent Document 2) has been established, which targets cloud gaming, a service that is operated via a network, and estimates the quality experienced by users during use from latency, bit rate, packet loss rate, etc.

[0005] "Joint Research Report on Improving Remote Control Technology for Construction Machinery," Public Works Research Institute, 2016. G.1072: Opinion model predicting gaming quality of experience for cloud gaming services, 2020, ITU-T

[0006] When remotely operating a work machine such as heavy machinery, it takes a certain amount of time for the operation to be reflected in the operation of the work machine due to factors such as the resistance of the control lever, the weight of the work machine, and hydraulic pressure. Therefore, a characteristic of this method is that delays have almost no effect on operability as long as they are below a certain delay time (Non-Patent Document 1). However, while Non-Patent Document 1 indicates the effect that changes in delay have on operability, it does not consider the overall effect of changes in delay and video quality on operability. Therefore, Non-Patent Document 1 cannot perform estimations that take into account the effect of changes in video quality on operability.

[0007] While Non-Patent Document 2 takes into account the overall impact of delay and video quality on operability, it fails to consider the impact of changes in delay on operability. Therefore, when attempting to estimate operability degradation due to delay based on Non-Patent Document 2, operability will be underestimated when delay is low. In Figure 1, the solid curve shows the actual operability versus delay. The dashed curve shows the relationship between delay and operability estimated based on Non-Patent Document 2. The "detectable delay" is the minimum amount of delay at which a difference in operability from when the delay is zero can be detected. Figure 1 shows that Non-Patent Document 2 estimates operability lower than the actual operability for delays less than the "detectable delay" that do not actually result in any degradation in operability.

[0008] Furthermore, the impact of latency and video quality on overall operability is such that when latency is small, degradation of video quality has a large impact on operability, whereas when latency is large, degradation of video quality has a small impact (Figure 2). However, in Non-Patent Document 2, the impact of degradation of video quality on operability is constant regardless of the magnitude of the latency. Therefore, the impact of latency and video quality on overall operability is not taken into consideration.

[0009] The present invention has been made in consideration of the above points, and has an object to make it possible to estimate operability for remotely operating a work machine, taking into account communication delays and video quality.

[0010] Therefore, in order to solve the above problem, the device has a delay operability estimation unit configured to calculate delayed operability, which is an estimated value of operability based on the delay, based on the characteristic that the greater the delay in communication related to operation during remote operation of a work machine, the lower the operability of the work machine, but that the operability does not decrease if the delay is less than a predetermined time; a video operability estimation unit configured to calculate video operability, which is an estimated value of operability based on the quality of the video, based on the characteristic that operability increases up to a certain bit rate with respect to communication of video of the work machine during remote operation, and the higher the packet loss rate, the lower the operability; and an integration unit configured to calculate an estimated value of operability of the work machine based on the delayed operability and the video operability.

[0011] It is possible to estimate the operability of remotely operating a work machine, taking into account communication delays and video quality.

[0012] FIG. 1 is a diagram for explaining the relationship between delay and operability. FIG. 2 is a diagram for explaining the influence of delay and video quality on operability. FIG. 3 is a diagram showing an example of the configuration of a remote operation system 20 in an embodiment of the present invention. FIG. 4 is a diagram showing an example of the hardware configuration of an operability estimation device 10 in an embodiment of the present invention. FIG. 5 is a diagram showing an example of the functional configuration of an operability estimation device 10 in an embodiment of the present invention. FIG. 6 is a flowchart for explaining an example of a processing procedure executed by the operability estimation device 10.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 3 is a diagram showing an example of the configuration of a remote operation system 20 according to an embodiment of the present invention. In Fig. 3, the remote operation system 20 is a computer system that enables a work machine 221, such as heavy machinery, located in a remote location 22 to operate from a remote operation room 21. An operation device 211 and a monitoring terminal 212 are arranged in the remote operation room 21. The work machine 221 and an imaging device 222 are arranged in the remote location 22.

[0014] The operation device 211 is a device that allows the worker to remotely operate the work machine 221. When the worker operates the operation device 211, the operation information is transmitted via the network 23, and the work machine 221 receives the operation information. The work machine 221 performs an operation in accordance with the operation information. In order to check the operation of the work machine 221 and the status of the work site from the remote control room 21, an imaging device 222 installed on the work machine 221 acquires (captures) video of the work machine 221. The captured video is encoded, and the encoded video (hereinafter referred to as "encoded video") is transmitted to the monitoring terminal 212 via the network 23. The monitoring terminal 212 displays video that has been decoded from the encoded video. The worker can check the work status by viewing the video.

[0015] The operability estimation device 10 is one or more computers that receive as input information related to video quality obtainable from the remote operation system 20 and information related to communication delays via the network 23, and outputs an estimated value of the operability of remote operation of the work machine 221 (hereinafter referred to as "operability o"). The operability estimation device 10 may be located outside the remote operation system 20, or may be located inside the remote operation system 20, such as in the remote operation room 21.

[0016] 4 is a diagram showing an example of the hardware configuration of an operability estimation device 10 according to an embodiment of the present invention. The operability estimation device 10 in FIG. 4 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, which are all interconnected via a bus B.

[0017] A program that realizes the processing in the operability estimation device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via the network 23. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0018] When an instruction to start the program is received, the memory device 103 reads out the program from the auxiliary storage device 102 and stores it. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the operability estimation device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to the network 23.

[0019] 5 is a diagram showing an example of the functional configuration of the operability estimation device 10 according to the embodiment of the present invention. In FIG. 5, the operability estimation device 10 includes an input unit 11, a delay operability estimation unit 12, a video operability estimation unit 13, an integration unit 14, and an output unit 15. Each of these units is realized by a process executed by a processor 104 of one or more programs installed in the operability estimation device 10.

[0020] The following describes the processing procedure executed by the operability estimation device 10. Fig. 6 is a flowchart for explaining an example of the processing procedure executed by the operability estimation device 10.

[0021] In step S101 , the input unit 11 receives input of information about video quality and information about delay from the remote control system 20 .

[0022] In the present embodiment, the bit rate of the encoded video and the packet loss rate during transmission of the encoded video are examples of information about video quality, and the delay time, which is the time from when an operation is input to the operation device 211 until it becomes visible on the video displayed by the monitoring terminal 212 that the work machine 221 has operated (strictly speaking, "the time until the work machine 221 starts operating in response to an operation input" + "the time until the video of the work machine 221 starting to operate is reflected on the monitor in the remote operation room 21"), is described as an example of information about delay. These bit rate, packet loss rate, and delay time are input to the operability estimation device 10.

[0023] The delay time may be measured as the difference between the start time of operation of the operating device 211 and the time when the image changes accordingly (for example, the difference between the start time of a lever operation to perform a turning operation and the time when the image begins to change due to the turning). Alternatively, the delay time may be the sum of the round trip time measurement results of a test packet such as a Ping on the network 23 (because there are two routes, from the operating device 211 to the work machine 221 and from the imaging device 222 to the monitoring terminal 212, for example, the average RTT of these two routes), the buffer time of each receiving terminal of the encoded image or operation information, and the processing time of the image encoding device and decoding device. In either case, the delay time is input as an average measurement time in units of one second. Methods other than these may also be used to input the delay time. The receiving terminal of the encoded image is the decoding device. The decoding device may be built into the monitoring terminal 212 or may be externally connected to the monitoring terminal 212. The receiving terminal of the operation information is the work machine 221. Similarly, the video encoding device may be built into the imaging device 222 or may be externally connected to the imaging device 222 .

[0024] The bit rate of the encoded video is determined using the amount of transmitted data of the encoded video packets and setting information obtained from the encoding device and decoding device in the remote control system 20. The amount of transmitted data can be calculated, for example, from the payload size of the video packet per unit time. The setting information is, for example, the bit rate targeted by the encoding device when encoding. In this case, the bit rate may be used as the bit rate of the encoded video.

[0025] The packet loss rate is determined by the ratio between the amount of transmitted data and the amount of received data of packets of encoded video, or by the packet loss rate per second calculated based on the results of transmitting and receiving test packets such as Ping on the network 23 that transfers encoded video within the remote control system 20. Note that other bit rates and packet loss rates may also be used.

[0026] Next, the delay operability estimation unit 12 calculates a delay operability o (delay operability) based on the delay time in the unit time interval (the average measurement time in 1-second units is the delay time) d [ms]. ｄｅｌａｙ In the following, the process for calculating the delay operability o from the delay time d [ms] is executed in steps S102 and S103. ｄｅｌａｙ A method for calculating .times. ...

[0027] Operability has a negative correlation with delay, and the larger the delay, the lower the operability. On the other hand, there is a characteristic (hereinafter referred to as "characteristic a") that the operator cannot detect the delay and the operability does not decrease if the delay is less than a predetermined time (hereinafter referred to as "detection limit t"). The delay operability estimation unit 12 takes this characteristic a into consideration.

[0028] First, in step S102, the delay operability estimation unit 12 selects a threshold value for the detection limit t. The value of t may be preset as a constant based on characteristics previously investigated through a subjective evaluation experiment or the like, or may be adjusted by the delay operability estimation unit 12 using an arbitrary coefficient taking into account the influence of video quality, or may be variable.

[0029] Next, the delay operability estimation unit 12 calculates the delay time d [ms] and the detection limit t, and if the delay time is less than t, it estimates the delay operability o, which is the same as when there is no delay. ｄｅｌａｙ If the detection limit t is exceeded, the delay operability o ｄｅｌａｙ The delay operability o is calculated by applying one of the following formulas (1-1) to (1-3), which is a model in which the delay operability o ｄｅｌａｙ , is calculated (S103).

[0030] In each formula, c １ , c ２ is a coefficient that differs for each equation. Note that other equations that take the characteristic a into consideration may also be used.

[0031] In addition, delay operability ｄｅｌａｙ , is a value that normalizes the delay-based operability to the range of 0-1.

[0032] Next, the video operability estimation unit 13 calculates a video operability o, which is an estimated value of operability based on the video quality (encoded video bit rate, packet loss rate), based on the bit rate b [kbps] of the encoded video and the packet loss rate p [%]. ｖｉｄｅｏ is calculated (S104).

[0033] Video quality correlates with bit rate, and the higher the bit rate, the higher the operability. On the other hand, when the bit rate reaches a certain level, the video quality reaches almost its upper limit, and the operability does not change even at bit rates above that level. In other words, the operability increases up to a certain bit rate. Therefore, using a model in which the operability increases as the bit rate increases, but the operability does not change above a certain bit rate, the video operability o when p = 0 is calculated. ｖｉｄｅｏ In addition, when packet loss occurs, the packet loss rate is negatively correlated with operability, and the larger the loss rate, the lower the operability. Also, in cases where the number of packet losses is more negatively correlated with operability than the packet loss rate, the higher the bit rate for the same packet loss rate, the greater the decrease in operability.

[0034] Therefore, the video operability estimation unit 13 takes into consideration the above characteristic (hereinafter referred to as "characteristic b"), and calculates the video operability o depending on the bit rate when the packet loss rate is 0 based on one of the following formulas (2-1) to (2-5). ｖｉｄｅｏ If the packet loss rate is greater than 0, the video operability o is calculated from the bit rate and packet loss rate. ｖｉｄｅｏ Calculate.

[0035] Here, in each formula, c ３ ~c ８ is a coefficient that differs for each formula. ｍａｘ is video operability ｖｉｄｅｏ is the maximum value that can be taken. ｖｉｄｅｏ The value range of o is 1-5. ｍａｘ The value of may be the maximum value of operability o obtained in a range of 1 to 5 in a prior subjective evaluation experiment, or may be 5, which is the theoretical maximum value of operability o.

[0036] Note that other equations may be used that take into account the characteristic b.

[0037] Next, the integration unit 14 calculates the delay operability o ｄｅｌａｙ and video operability ｖｉｄｅｏ Based on the above, the operability o of the entire remote operation system 20 is calculated based on both the delay and the video quality (S105).

[0038] Operability o is delayed operability o ｄｅｌａｙ , as shown in Figure 2, there is a characteristic (hereinafter referred to as "characteristic c") in which operability o decreases synergistically due to the influence of a decrease in video quality. Here, "synergistic" does not mean that operability o decreases while maintaining the difference in video quality relative to the overall delay, but rather expresses a tendency in which, when delay decreases due to the influence of video quality, the operability o of the delay does not decrease even if the video quality decreases in the case of a relatively large delay, but decreases when the video quality decreases in the case of a relatively small delay. In other words, "synergistic" expresses the tendency that when the delay is small, the difference in video quality is reflected in operability o, but once operability o relative to the delay begins to decrease, the slope of the decrease in operability o changes depending on the video quality.

[0039] The integration unit 14 is ｖｉｄｅｏ Decreased and delayed operability ｄｅｌａｙ Taking into account the synergistic effect of the decrease in the operability o, the operability o is calculated based on the following formula (3-1) or formula (3-2).

[0040] Here, in each formula, c ９ is a coefficient that differs for each formula. ｍａｘ is as described above. The value of operability o ranges from 1 to 5, and the larger the value, the higher the operability. In both equations, the video operability o ｖｉｄｅｏ and delay operability ｄｅｌａｙ The formula includes a term in which the characteristic c is multiplied by the characteristic c. This allows the above synergistic effects to be reflected in each formula. Note that other formulas may be used that take the characteristic c into consideration.

[0041] Next, the output unit 15 performs output based on the operability o (S106). For example, the output unit 15 may output (display) the operability o to a predetermined terminal. Alternatively, the output unit 15 may output control information for the remote operation system 20. For example, if the operability o is equal to or less than a certain value, the output unit 15 may transmit control information to the remote operation system 20 to stop the remote operation system 20.

[0042] The value of t and the values ​​of each coefficient are derived by optimizing the input and the results of a subjective evaluation of the operability for the input using a method such as the least squares method. The value of t and the values ​​of each coefficient may be derived by a method other than the above-mentioned derivation method, or predetermined coefficients may be used.

[0043] As described above, according to this embodiment, it is possible to estimate the operability o taking into consideration communication delays and video quality for the remote operation of the work machine 221. By monitoring the operability o, if the operability o continues to drop to a certain value, it becomes possible to notify the operator in the remote control room 21 to stop operation, to make an emergency stop of the work machine 221, or to notify the remote operation manager, thereby contributing to the prevention of accidents and the like caused by a drop in the operator's operability.

[0044] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0045] REFERENCE SIGNS LIST 10 Operability estimation device 11 Input unit 12 Delay operability estimation unit 13 Video operability estimation unit 14 Integration unit 15 Output unit 20 Remote operation system 21 Remote operation room 22 Remote location 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device 211 Operation device 212 Monitoring terminal 221 Work machine 222 Imaging device B Bus

Claims

1. A device comprising: a delay operability estimation unit configured to calculate delay operability, which is an estimate of operability based on delay, based on the characteristic that the greater the delay in communication related to operation during remote operation of a work machine, the lower the operability of the work machine, but that the operability does not decrease if the delay is less than a predetermined time; a video operability estimation unit configured to calculate video operability, which is an estimate of operability based on the quality of the video, based on the characteristic that, with regard to communication of video of the work machine during remote operation, operability increases up to a certain bit rate, and the higher the packet loss rate, the lower the operability; and an integration unit configured to calculate an estimate of the operability of the work machine based on the delay operability and the video operability.

2. The device according to claim 1, wherein the video operability estimation unit is further configured to calculate the video operability based on the characteristic that the higher the bit rate for the same packet loss rate, the greater the deterioration in operability.

3. The device according to claim 1, characterized in that the integrating unit is configured to calculate an estimated value of the operability of the work machine based on the characteristic that when the operability of the work machine in response to the delay begins to deteriorate, the slope of the deterioration in the operability varies depending on the quality of the video.

4. A device that calculates an estimated value of the operability of a work machine based on communication delay time related to operation in remote control of the work machine, and video bit rate information and video packet loss rate related to video of the work machine in said remote control.

5. A operability estimation method, characterized in that a computer executes the following steps: a delay operability estimation procedure for calculating delay operability, which is an estimate of operability based on delay, based on the characteristic that the greater the delay in communication related to operation during remote operation of a work machine, the lower the operability of the work machine, but that the operability does not decrease if the delay is less than a predetermined time; a video operability estimation procedure for calculating video operability, which is an estimate of operability based on the quality of the video, based on the characteristic that, with regard to communication of video of the work machine during remote operation, operability increases up to a certain bit rate, and the higher the packet loss rate, the lower the operability; and an integration procedure for calculating an estimate of the operability of the work machine based on the delay operability and the video operability.

6. A program that causes a computer to execute the following steps: a delay operability estimation procedure that calculates delayed operability, which is an estimate of operability based on delay, based on the characteristic that the greater the delay in communication related to operation during remote operation of a work machine, the lower the operability of the work machine, but that the operability does not decrease if the delay is less than a predetermined time; a video operability estimation procedure that calculates video operability, which is an estimate of operability based on the quality of the video, based on the characteristic that operability increases up to a certain bit rate with respect to communication of video of the work machine during remote operation, and the higher the packet loss rate, the lower the operability; and an integration procedure that calculates an estimate of the operability of the work machine based on the delayed operability and the video operability.

Images