Remote machine control device and program, and remote machine control system

Abstract

To provide a remote machine operating device, program, and remote machine operating system that enable a user to operate a machine installed in a remote location without feeling uncomfortable while monitoring a video of the machine's movement. [Solution] The remote machine control device 10 is used by an operator to control a work machine at a remote location while viewing monitoring footage, and is equipped with a future control information prediction unit 25 that predicts future control information, which is future control information after a time equivalent to the transmission delay time between the work machine 80 and the operation time at the operation unit 11 has elapsed, based on control information obtained from an operation unit 11 operated by the operator, specific information related to the operator's operation, and a future prediction model, and a transmission unit 26 that transmits the future control information predicted by the future control information prediction unit 25 to the work machine.

Description

[Technical field]

[0001] The present invention relates to a remote machine control device and program, and a remote machine control system. [Background technology]

[0002] Conventionally, there is known a remote machine operation system in which an operator remotely controls a machine installed in a remote location via a network. Specifically, the operator operates the machine by operating a control lever at hand while monitoring a video of the machine's movement. In such a remote machine operation system, there is a problem that the operator feels a discrepancy between the monitoring video and the actual lever operation due to a transmission delay in the network.

[0003] In response to this, a technique has been disclosed for suppressing the effect of communication delays between an operation terminal (operator) and an actual robot (machine) on the operation of the operator (see Patent Document 1).

[0004] The technology in Patent Document 1, when monitoring video of the movements of an actual robot operating in a remote location with a delay due to communication delay, models the actual robot and its surrounding objects, synthesizes predicted images that compensate for communication delays based on these models, and displays these as monitoring images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-26028 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology of Patent Document 1, the displayed monitoring image depends heavily on the accuracy of the model that is the basis of the predictive synthesis. Meanwhile, since the surroundings of the real robot are not necessarily specific objects, it is difficult to always model the surroundings accurately, resulting in a problem that appropriate monitoring images cannot be displayed.

[0007] On the other hand, some of the inventors of the present application have proposed a technology to acquire myoelectric potential information from an electromyographic sensor attached to the operator's body and use that myoelectric potential information to predict the operation information of a machine located in a remote location (Patent Application No. 2022-154191).

[0008] The above technique requires the operator to wear an electromyogram sensor on his or her body in order to obtain electromyogram information, which is troublesome for the operator.

[0009] The present invention has been proposed in consideration of the above-mentioned situation, and provides a remote machine control device, program, and remote machine control system that enable a user to operate a machine installed in a remote location without feeling uncomfortable while monitoring video of the machine's movement. [Means for solving the problem]

[0010] The remote machine control device according to the present invention allows an operator to control the machine while watching a monitoring image. Operate the control panel A remote machine control device for controlling a work machine at a remote location, comprising: an operation force acquisition unit that acquires an operation force applied to the operation unit; and a future prediction model storage unit that stores a future prediction model, which is a machine learning model trained using, as teacher data, operation information at a time of operation on the operation unit, operation information at a time after a transmission delay time between the operation time and the work machine device, and time series data of the operation force within a predetermined range in the past from the operation time; The operator operates the The above Operation information obtained from an operation unit, The operating force acquired by the operating force acquisition unit and, The future prediction model stored in the future prediction model storage unit The system is equipped with a future steering information prediction unit that predicts future steering information, which is future steering information after a time equivalent to the transmission delay time between the work machine device and the operation unit has elapsed based on a future prediction model and the operation time of the operation unit as a reference, and a transmission unit that transmits the future steering information predicted by the future steering information prediction unit to the work machine device.

[0011] The program according to the present invention causes a computer to function as each part of the remote machine control device.

[0012] The remote machine control system of the present invention comprises the remote machine control device and the work machine device that receives the future operation information transmitted from the remote machine control device and transmits the monitoring image to the remote machine control device. Effect of the Invention

[0013] The present invention reduces the effects of transmission delays between machines installed in remote locations, allowing an operator to operate the machine without feeling uncomfortable while viewing the monitoring video. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic diagram of a remote machine control system according to a first embodiment. FIG. [Diagram 2] 2 is a block diagram showing the functional configuration of a remote machine control device and a work machine device. FIG. [Diagram 3] FIG. 2 is a schematic diagram illustrating an example of an operation unit. [Figure 4] 4 is a block diagram showing the functional configuration of a future driving information prediction unit; [Diagram 5] 10 is a flowchart showing a prediction processing routine performed by a driving information prediction unit. [Figure 6] 1 is a flowchart showing the machine learning process of a future prediction model. [Figure 7] 4 is a waveform diagram showing an example of an operating force (operation information) obtained from an operating unit. FIG. [Figure 8] FIG. 4 is a schematic diagram showing a comparison between operation information (operational force) and future operation information. [Figure 9] FIG. 4 is a comparison diagram between actual operation information (operational force) and future operation information. [Figure 10] FIG. 11 is a block diagram showing a functional configuration of a future driving information prediction unit according to a second embodiment. [Figure 11]10 is a flowchart showing a prediction processing routine performed by a driving information prediction unit according to the second embodiment. [Figure 12] 13 is a flowchart showing a model update routine which is a subroutine of step S7. [Figure 13] FIG. 11 is a block diagram showing the functional configuration of a remote machine control device and a work machine device according to a third embodiment. [Figure 14] FIG. 13 is a block diagram showing the functional configuration of a remote machine control device and a work machine device according to a fourth embodiment. [Figure 15] 4 is a waveform diagram showing an example of myoelectric potential information acquired by a myoelectric potential information acquisition unit. FIG. [Figure 16] FIG. 13 is a waveform diagram showing an example of each element of an extracted muscle synergy time pattern vector. [Figure 17] 11 is a diagram comparing actual operation information and future operation information with and without myoelectric potential information. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0016] [First embodiment] (system configuration diagram) FIG. 1 is a schematic diagram of a remote machine control system 1 according to a first embodiment. The remote machine control system 1 includes a remote machine control device 10 that is operated by an operator, and a work machine device 80 that performs a predetermined task while communicating with the remote machine control device 10 via a network 50.

[0017] The work machine 80 is located at a location, for example, a remote location, away from the remote machine control device 10. The work machine 80 may be fixed to the location, or may be movable from the location.

[0018] The camera 84 of the work machine device 80 is installed in a position where it can capture an image of the working unit 83 performing a predetermined task. The camera 84 captures the work status of the working unit 83 to generate monitoring video. The work machine device 80 transmits the generated monitoring video to the remote machine control device 10 via the network 50.

[0019] The monitor 14 of the remote machine control device 10 displays the monitoring video transmitted from the work machine 80, i.e., the work status of the working unit 83. While viewing the monitoring video displayed on the monitor 14, the operator operates the operation unit 11 of the remote machine control device 10 to operate the work machine 80 (including the working unit 83 included therein) located at a remote location. In this embodiment, the operator's movement of the operation unit 11 is referred to as "operation," and the movement of the work machine 80 (including the working unit 83) is referred to as "operation."

[0020] In this way, when the operator operates the operation unit 11, the remote machine control device 10 transmits control information to the work machine device 80. Then, when the work machine device 80 receives the control information, it performs an operation corresponding to the operation of the operation unit 11.

[0021] When information is transmitted between the remote machine control device 10 and the work machine 80, a transmission delay occurs due to the network 50. The transmission delay time associated with the information transmission from the remote machine control device 10 to the work machine 80 (outbound) is assumed to be Δ1.

[0022] When the remote machine control device 10 transmits steering information at time t1 and the work machine device 80 receives the steering information at time t2, the following equation holds: t2=t1+Δ1

[0023] Meanwhile, the transmission delay time associated with the information transmission from the work machine 80 to the remote machine control device 10 (return route) is denoted as Δ2. When the work machine 80 transmits monitoring video of the work status of the working unit 83 captured by the camera 84 at time t'2, and the remote machine control device 10 receives and displays the monitoring video at time t3, the following equation holds: t3=t'2+Δ2

[0024] The return path transmission delay time Δ2 includes not only the delay caused by the information transmission on the network 50, but also the delay caused by the image processing of the monitoring video performed before and after the information transmission. As a result, the transmission delay time Δ1 and the transmission delay time Δ2 do not necessarily match.

[0025] Here, assuming that t2=t'2, t3=t1+Δ1+Δ2. In other words, when the operator operates the operation unit 11 of the remote machine control device 10 at time t1, the monitor 14 displays the working status of the working unit 83 due to that operation at time t3.

[0026] FIG. 2 is a block diagram showing the functional configuration of the remote machine control device 10 and the work machine device 80. The remote machine control device 10 includes an operation unit 11 operated by the user, a control information prediction unit 12 that predicts future control information, a video receiving unit 13 that receives monitoring video, and a monitor 14 that displays the work status of the work machine device 80 as monitoring video.

[0027] 3 is a schematic diagram showing an example of the operation unit 11. There is no particular limitation on the operation unit 11 as long as it is something that can be operated by an operator, but in this embodiment, it is an operation lever. The operation unit 11 includes a stick unit 11s for operating the work machine device 80, and a dial adjustment unit 11d for manually adjusting a predicted time width Δ, which will be described later. The predicted time width Δ adjusted by the dial adjustment unit 11d is used by the operation information prediction unit 12.

[0028] The stick portion 11s is formed in a rod shape, and stands upright perpendicular to the base when it is not performing any action (neutral state). The stick portion 11s can be tilted in any direction in the xy plane in response to an operator's operation. The angle at which the stick portion 11s is tilted (displaced) in any direction from the neutral state is called the displacement angle θ, and the projection of this onto the x-axis and y-axis, i.e., the x component and y component, are called the displacement angle θx and the displacement angle θy, respectively.

[0029] A spring-like member that can expand and contract in the x and y directions is attached to the base of the stick 11s. When the stick 11s is tilted by a displacement angle θ from the neutral state by the operator's operation, the spring-like member applies forces to the stick 11s that are simply proportional to the displacement angles θx and θy, respectively, to return the stick 11s to the neutral state. The stick 11s is also provided with sensors that output signals according to the expansion of each spring-like member.

[0030] When the stick unit 11s is tilted by a displacement angle θ by the operation of the operator, the operation unit 11 supplies a signal corresponding to the displacement angle θ from the neutral state and a signal corresponding to the extension of each spring-like member in the x-direction and y-direction to the operation information prediction unit 12. Hereinafter, any time when the operator continuously operates the operation unit 11 is defined as an operation time t.

[0031] As shown in FIG. 2, the driving information prediction unit 12 includes a displacement information acquisition unit 21, a calculation unit 22, an operating force acquisition unit 23, a transmission delay time measurement unit 24, a future driving information prediction unit 25, and a transmission unit 26.

[0032] The displacement information acquisition unit 21 receives a signal corresponding to a displacement angle θ from the neutral state of the operation unit 11, and detects the displacement angle θ for each sampling period. The displacement information acquisition unit 21 supplies the detected displacement angle θ to the calculation unit 22.

[0033] The calculation unit 22 calculates the steering information ξ(θ) using the displacement angle θ supplied from the displacement information acquisition unit 21 as an input for each sampling period, and supplies the steering information ξ(θ) to the future steering information prediction unit 25. The steering information ξ(θ) is information for steering the work machine device 80, and is expressed as a predetermined function of the displacement angle θ.

[0034] The operating force acquisition unit 23 receives a signal corresponding to the expansion in the x and y directions of a spring-like member provided in the operating unit 11, acquires the operating force F applied to the operating unit 11 by the operator for each sampling period, and supplies this to the future driving information prediction unit 25. In the following, the operating force applied in the x direction is designated as Fx, and the operating force applied in the y direction is designated as Fy.

[0035] The transmission delay time measurement unit 24 measures a predicted time width Δ when predicting future steering information. The predicted time width Δ is the sum of the transmission delay time Δ1 associated with the information transmission on the outbound route described above and the transmission delay time Δ2 associated with the information transmission on the return route described above.

[0036] The method of measuring the predicted time width Δ is not particularly limited as long as it can be measured by the transmission delay time measurement unit 24. The transmission delay time measurement unit 24 may provide a transmission delay time Δmeasure (fixed value) measured in advance in the preparation stage as the predicted time width Δ. Alternatively, the predicted time width Δ may be measured using an "echo request / echo reply" message of the Internet Control Message Protocol (ICMP), which is used in the Ping command that checks the connection status in the network.

[0037] Also, as described above, the transmission delay time Δ1 and the transmission delay time Δ2 do not necessarily match. Therefore, for example, if the internal clocks of the remote machine control device 10 and the work machine 80 are synchronized with high precision (for example, on the order of sub-milliseconds) by sharing GPS information, the following method is possible. That is, the transmission delay time measurement unit 24 may calculate the transmission delay times Δ1 and Δ2 using the transmission time and reception time (of information) recorded by the remote machine control device 10 and the work machine 80, respectively, and then measure the predicted time width Δ by taking the sum of these values.

[0038] However, if the predicted time width Δ measured as described above is used as is to predict the operation information, the work machine device 80 (for example, the working unit 83) may appear to react too sensitively to the operator's operation on the monitoring image, which may cause the operator to feel uncomfortable. In order to suppress such a sense of discomfort, when the operator operates the adjustment dial 11d of the operation unit 11 shown in Figure 3, the transmission delay time measurement unit 24 may adjust the size of the predicted time width Δ in accordance with the amount of operation.

[0039] The future steering information prediction unit 25 predicts, for each sampling period, future steering information ξp(t+Δ), which is steering information after a first predetermined time has elapsed from the operation time t, based on the steering information ξ(t) as a function of the displacement angle θ(t) at the operation time t. The first predetermined time is equal to the predicted time width Δ measured by the transmission delay time measurement unit 24.

[0040] Specifically, the future operation information prediction unit 25 predicts future operation information ξp(t+Δ), which is the operation information after a first predetermined time has elapsed from the operation time t, by referring to past operation forces F (multiple sampling data) within the period from the operation time t to a second predetermined time before the operation time t, for the operation information ξ(t) at the operation time t.

[0041] The reason why the future steering information prediction unit 25 can predict the future steering information ξp(t+Δ) by using the sampling data of the past operating force F is as follows. As described above, the steering information ξ(t) is a function of the displacement angle θ(t) of the operation unit 11. The displacement angle θ(t) is determined by the balance between the force applied by the operator to the operation unit 11 and the force caused by the spring-like member that the operator feels as a response from the operation unit 11. For this reason, the future steering information prediction unit 25 can predict the future steering information ξp(t+Δ) with high accuracy by using the response of the operation unit 11 up to that point, that is, the sampling data of the past operating force F. Specifically, the future steering information prediction unit 25 is configured as shown in FIG. 4.

[0042] FIG. 4 is a block diagram showing the functional configuration of future steering information prediction unit 25. As shown in FIG. The future steering information prediction unit 25 includes a future prediction model holding unit 31 that holds a future prediction model, a prediction unit 32 that predicts future steering information ξp(t+Δ), and a steering information holding unit 33 that holds the steering information ξ(t) and future steering information ξp(t+Δ) at operation time t.

[0043] The future prediction model held in the future prediction model holding unit 31 is, for example, a machine learning model that has been externally trained by machine learning. The future prediction model is trained by machine learning using a large amount of teacher data, and thereby constructs a relationship between the operation information ξ(t) at the operation time t, the past operation force F (multiple sampling data) within the period from the operation time t to a second predetermined time before, and the future operation information ξp(t+Δ).

[0044] The prediction unit 32 predicts future steering information ξp(t+Δ) using the future prediction model held in the future prediction model holding unit 31. Specifically, the prediction unit 32 inputs the steering information ξ(t) at operation time t supplied from the calculation unit 22 shown in Fig. 2 and the past operating force F = (Fy, Fy) (multiple sampling data within a period from the operation time t to a second predetermined time before) supplied from the operating force acquisition unit 23 to the future prediction model. Then, the prediction unit 32 outputs the information output from the future prediction model as future steering information ξp(t+Δ).

[0045] The steering information holding unit 33 holds the steering information ξ(t) at the operation time t supplied from the calculation unit 22 and the future steering information ξp(t) predicted by the prediction unit 32 in association with each other.

[0046] 2, the future steering information prediction unit 25 supplies the predicted future steering information ξp(t+Δ) to the transmission unit 26. The transmission unit 26 transmits the future steering information ξp(t+Δ) predicted by the future steering information prediction unit 25 to the work machine device 80 via the network 50.

[0047] In this way, the remote machine control device 10 transmits the future control information ξp(t+Δ) to the work machine device 80 at the operation time t. Here, the future control information ξp(t+Δ) refers to control information predicted to be generated by the operator in the future after the predicted time width Δ (=Δ1+Δ2) has elapsed from the operation time t.

[0048] The future operation information ξp(t+Δ) (=ξp(t+Δ1+Δ2)) transmitted from the remote machine operation device 10 is delayed by time Δ1 when passing through the network 50. Therefore, the future operation information received by the work machine device 80 at the operation time (t+Δ1) is ξp(t+Δ2).

[0049] As shown in Figure 2, the work machine device 80 is equipped with a receiving unit 81 that receives future operation information ξp(t+Δ) from the remote machine control device 10, a control unit 82 that generates machine control information from the received operation information, a working unit 83 that performs a specified task in accordance with the machine control information generated by the control unit 82, a camera 84 that captures the work status of the working unit 83 and generates a monitoring image, and an image transmitting unit 85 that transmits the monitoring image of the working unit 83 obtained by the camera 84.

[0050] The control unit 82 controls the working unit 83 using machine control information generated based on the future operation information ξp(t+Δ) received by the receiving unit 81. As a result, the working unit 83 performs a predetermined task based on the future operation information ξp(t+Δ2) after the time Δ1 has elapsed since the operation time t. The camera 84 captures the working status of the working unit 83. The video transmission unit 85 performs predetermined image processing, such as video compression processing, on the monitoring video of the working unit 83 captured by the camera 84, and transmits the obtained image-processed monitoring video to the remote machine control device 10.

[0051] The monitoring video transmitted from the work machine 80 arrives at the remote machine control device 10 with a delay of time Δ2 when passing through the network 50. The video receiving unit 13 of the remote machine control device 10 receives the monitoring video transmitted from the work machine 80, performs predetermined image processing on the video, and supplies the video to the monitor 14. The monitor 14 displays the monitoring video supplied from the video receiving unit 13.

[0052] The future operation information ξp(t+Δ1+Δ2) transmitted from the remote machine control device 10 arrives at the work machine 80 with a delay of time Δ1 when passing through the network 50. As a result, the working unit 83 of the work machine 80 performs a predetermined task based on the future operation information ξp(t+Δ2) with respect to the operation time t of the operation unit 11, that is, a task for a time Δ2 in the future from the operation time t. Then, the monitoring video of the working unit 83 captured by the camera 84 is displayed on the monitor 14 of the remote machine control device 10 with a delay of time Δ2 when passing through the network 50.

[0053] As a result, the monitor 14 displays a monitoring image of a predetermined task based on the future operation information ξp(t) at the operation time t of the operation unit 11, i.e., a task status predicted to be performed at the same time as the operation time t. Therefore, the operator can operate the operation unit 11 without being affected by the transmission delay of the network 50, in other words, without feeling uncomfortable about the monitoring image displayed on the monitor 14.

[0054] FIG. 5 is a flowchart showing a prediction processing routine performed by the driving information predicting unit 12. In step S1, the displacement information acquisition unit 21 detects whether or not the operation unit 11 has been operated. If the operation unit 11 has been operated, the process proceeds to the next step.

[0055] In step S2, the displacement information acquisition unit 21 acquires the displacement angle θ(t) of the operation unit 11 at the operation time t. The calculation unit 22 generates operation information ξ(t) based on a predetermined function to which the displacement angle θ(t) acquired by the displacement information acquisition unit 21 is input, and supplies the obtained operation information ξ(t) to the future operation information prediction unit 25.

[0056] The operating force acquisition unit 23 acquires the operating force applied to the operating unit 11 through a sensor provided on a spring-like member at the base of the stick portion 11s of the operating unit 11, and supplies the operating force to a future driving information prediction unit 25.

[0057] In step S3, the transmission delay time measurement unit 24 measures a predicted time span Δ corresponding to a transmission delay time caused by round-trip information transmission between the remote machine control device 10 and the work machine device 80. The predicted time span Δ may be an actual measurement value measured by the above-mentioned conventional technology, or may be a preset fixed value.

[0058] In step S4, the future operation information prediction unit 25 predicts future operation information ξp(t+Δ), which is the operation information after a first predetermined time has elapsed from the operation time t, by referring to past operation forces F (multiple sampling data) within the period from the operation time t to a second predetermined time before the operation time t, for the operation information ξ(t) at the operation time t supplied from the calculation unit 22.

[0059] In step S5, the transmission unit transmits the future steering information ξp(t+Δ) predicted by the future steering information prediction unit 25 to the work machine device 80.

[0060] In step S6, it is determined whether or not the operation of the work machine 80 has ended, and if it is determined that the operation has not ended, the process returns to step S1. If it is determined that the operation of the work machine 80 has ended, the series of processes ends.

[0061] Next, the machine learning process of the future prediction model will be described. The future prediction model is a machine learning model trained using the operation information ξ(t) and the operation force F(t) at the operation time t that are actually measured in advance as training data.

[0062] (Learning process for future prediction model) 6 is a flowchart showing the machine learning process of the future prediction model. That is, a machine learning device (not shown) performs the machine learning process of the future prediction model according to the steps from step S11 onwards.

[0063] In step S11, the machine learning device sets an initial value w to the operation time ta, and acquires the operation information ξ(ta), where w is the above-mentioned second predetermined time.

[0064] In step S12, the machine learning device acquires past operation forces (F(ta-w), . . . , F(ta)) that are sampled and actually measured within a range from the operation time ta to a time before w.

[0065] In step S13, the machine learning device uses a future prediction model with the steering information ξ(ta) and past operating forces (F(ta-w),...,F(ta)) as input to calculate (predict) future steering information ξp(ta+Δ), which is the steering information at the operation time (ta+Δ) that is a first predetermined time Δ after the operation time ta.

[0066] In step S14, the machine learning device acquires operation information ξ(ta+Δ) which is an actual measurement value at the operation time (ta+Δ).

[0067] In step S15, the machine learning device updates the internal parameters of the future prediction model in a direction that reduces the difference between the future steering information ξp(ta+Δ) and the corresponding steering information ξ(ta+Δ), which is an actual measured value as teaching data.

[0068] In step S16, the machine learning device increases the operation time ta by a time δ, where δ is the sampling time interval.

[0069] In step S17, the machine learning device ends the process if the measured value ξ(ta) of the operation information at the operation time ta does not exist, and returns to step S12 if the measured value ξ(ta) exists.

[0070] As described above, in the remote machine operating system 1 according to this embodiment, the remote machine operating device 10 predicts and transmits future operation information, which is operation information for the future, by an amount corresponding to a round-trip transmission delay time, to the work machine 80 installed at a remote location. This allows the operator to operate the work machine 80 without feeling uncomfortable while checking the monitoring video of the work status of the work machine 80.

[0071] (Measurement example) Next, the calculation (prediction) of the future steering information ξp(t+Δ) using the actual measured value of the steering information will be described with a concrete example. Although the control information ξ has been expressed as a predetermined function of the displacement angle θ, in the following, in order to reduce the computational load, it is assumed to be simply proportional to the displacement angle θ, that is, a linear function of the displacement angle θ. Meanwhile, as described above, the operating force F is also simply proportional to the displacement angle θ. Therefore, when the constant k is used, the following relationship is established between the control information ξ and the operating force F. ξ=k F

[0072] In this embodiment, k=1 is set to further simplify the calculation. In other words, the operation force F is used as the steering information ξ as it is. Therefore, the steering information ξ(t) at time t is equivalent to the steering force F(t), and its x-component and y-component are also expressed as Fx(t) and Fy(t), respectively. The x-component and y-component of the future steering information ξp(t+Δ), which is the steering information after the predicted time width Δ from the time t, are also Fpx(t+Δ) and Fpy(t+Δ), respectively. Here, the x-component and y-component of the steering information ξ (operation force F) are each capable of being handled independently of each other.

[0073] FIG. 7 is a waveform diagram showing an example of the operating force F (operation information ξ) obtained from the operating unit 11. As shown in FIG. The vertical axis indicates values ​​normalized in the range of -1.0 to 1.0 based on the maximum absolute value of the x component (Fx) and y component (Fy) of the operating force F output in sample units from the operating unit 11. The horizontal axis indicates the time axis (seconds). Each thick solid line is a plot of the first 1000 samples (20 seconds worth) sampled at a sampling frequency of 50 Hz.

[0074] In the following, we will mainly explain how to handle Fx, which is the x-component of the control information ξ, but the y-component Fy can also be handled in a similar manner.

[0075] FIG. 8 is a diagram that shows a schematic comparison between the operation information ξ (operation force Fx) obtained by actual measurement and the future operation information ξp (future operation force Fpx) predicted using the same. The future steering information ξp(t+Δ) (future operating force Fpx(t+Δ)) is predicted from the steering information ξ(ta) (operating force Fx(ta)) at the operation time ta and sample data of the past operating force F within the period from the operation time ta to the time w before (Fx(ta-w), . . . , Fx(ta)). Note that in this embodiment, since the steering information ξ is equivalent to the operating force Fx, Fx(ta) is the actually measured steering information ξ(ta) at the operation time ta and is also one of the sample data of the past operating force F used to predict ξp(t+Δ).

[0076] 9 is a comparative diagram between the actual operation information ξ (operation force F) and the future operation information ξp (future operation force Fp) obtained by an experiment. Here, the prediction time width Δ, which is the first predetermined time described above, is set to 500 milliseconds, and the actual measured values ​​of the operation forces Fx(t) and Fy(t) are compared with the future operation forces Fpx(t) and Fpy(t) predicted using the future prediction model.

[0077] In the experiment shown in FIG. 9, a machine learning model that has been subjected to the machine learning processing shown in FIG. 6, specifically, LSTM (Long Short Term Memory), was adopted as the future prediction model.

[0078] In addition, the acquisition range period w for the past sampling data of operating forces (Fx(tw),...,Fx(t)) and (Fy(tw),...,Fy(t)), which corresponds to the second predetermined period described above, was set to w = 2.0 seconds.

[0079] Furthermore, the sampling frequency of the actual control information ξ (operating force F) was set to 50 Hz. Therefore, the past operating force F required to predict the future control information ξp (future operating force Fp) is sampled with a time width of 20 milliseconds within the acquisition range period of 2 seconds, and as a result, 100 samples are obtained as the input past operating force F.

[0080] This indicates that, starting from a certain operation time t, the future operation information ξp 0.5 seconds after the operation time t is predicted using the operation force F sampled for the past 2.0 seconds. Therefore, as shown by the dotted line in Fig. 9, for example, the future operation force Fpx(t) first appears at the operation time t = 2.5 seconds.

[0081] In this embodiment, LSTM is used as an example of a machine learning model, but there is no particular limitation as long as the machine learning model predicts time-series data. For example, the machine learning model may be a GRU (Gated Recurrent Unit) or a Time Convolution Network, which are recurrent neural networks.

[0082] In this embodiment, the x component and the y component of the operation information ξ (operating force F) are not dependent on each other and can be handled independently.

[0083] However, when interaction between the x and y components of the control information ξ (operating force F) is expected, machine learning processing may be performed on the machine learning model for calculating Fpx(t+Δ) by using not only the x component of the past operating force vector (Fx(tw),...,Fx(t)) but also its y component (Fy(tw),...,Fy(t)) as training data.

[0084] 4 stores the steering information ξ(t) at the operation time t and the future steering information ξp(t) predicted by the prediction unit 32 in association with the operation time t. The information stored in the steering information storage unit 33 may be retrieved to the outside of the system at a predetermined timing, and the retrieved information may be used as teacher data or the like to execute machine learning processing and update the internal parameters of the future prediction model. In this case, the future prediction model stored in the future prediction model storage unit 31 may be replaced with the future prediction model updated as described above under predetermined conditions.

[0085] [Second embodiment] Next, a second embodiment will be described. The same components as those in the first embodiment are denoted by the same reference numerals, and duplicated descriptions will be omitted. In the second embodiment, a future steering information prediction unit 25A shown in the following FIG. 10 is used instead of the future steering information prediction unit 25 shown in FIG. 4.

[0086] 10 is a block diagram showing a functional configuration of a future steering information prediction unit 25A according to the second embodiment. The future steering information prediction unit 25A is obtained by adding a future prediction model update unit 34 to the configuration of the future steering information prediction unit 25 shown in FIG. The future prediction model update unit 34 compares the operation information ξ(t) at operation time t supplied from the calculation unit 22 with future operation information ξp(t) predicted at the operation time (t-Δ) and stored in the operation information storage unit 33, and updates the future prediction model stored in the future prediction model storage unit 31.

[0087] 11 is a flowchart showing a prediction processing routine by the driving information prediction unit 12A (not shown) including the future driving information prediction unit 25A. Here, the prediction processing routine executes steps S1 to S5 in the same manner as in FIG. 5, then executes a new step S7, and finally executes step S6.

[0088] In step S7, the future prediction model update unit 34 compares the driving information ξ(t) with the future driving information ξp(t) to update the future prediction model held in the future prediction model holding unit 31. Specifically, the model update routine shown in FIG. 12 is executed.

[0089] FIG. 12 is a flowchart showing a model update routine which is a subroutine of step S7 shown in FIG. In step S21, the future prediction model update unit 34 acquires the operation information ξ(t) at the operation time t supplied from the calculation unit 22.

[0090] In step S22, the future prediction model update unit 34 obtains future operation information ξp(t) at operation time t predicted at operation time (t-Δ), which is a prediction time width Δ in the past from the operation time t, from the previously predicted future operation information stored in the operation information storage unit 33.

[0091] In step S23, the future prediction model update unit 34 updates the internal parameters of the future prediction model held in the future prediction model holding unit 31 so as to reduce the difference between the operation information ξ(t) and the future operation information ξp(t). Then, the future prediction model update unit 34 repeatedly executes steps S21 to S23 each time the operation time t advances.

[0092] However, if the future prediction model update unit 34 were to update the future prediction model every time sample data of the steering information ξ(t) and the future steering information ξp(t) is supplied, large-scale and high-speed computing power would be required, resulting in a large computational load. Therefore, the future prediction model update unit 34 may update the future prediction model for each sample data of the steering information ξ(t) and the future steering information ξp(t) every predetermined number of samples (e.g., 20 samples).

[0093] As described above, the remote machine control device 10 of this embodiment updates the future prediction model from time to time in parallel with predicting future operation information so that the previously predicted future operation information matches the actual operation information. Therefore, even if a change occurs in the work situation of the work machine device 80 and the prediction accuracy of the future operation information decreases, the discomfort felt by the operator who is controlling the work machine device 80 while checking the monitoring video of the work situation can be gradually reduced.

[0094] [Third embodiment] Next, a third embodiment will be described. Note that the same reference numerals are used to designate the same parts as those in the above-mentioned embodiment, and duplicated explanations will be omitted.

[0095] Fig. 13 is a block diagram showing the functional configuration of a remote machine control device and a work machine according to the third embodiment. The remote machine control system 1B is equipped with a work machine 80B instead of the work machine 80 shown in Fig. 2. The work machine 80B is obtained by adding a predicted image synthesis unit 86 that synthesizes a predicted image to the configuration of the work machine 80 shown in Fig. 2.

[0096] In the first embodiment, the remote machine control device 10 predicts future control information ξp(t+Δ), which is control information after a predicted time width Δ (=Δ1+Δ2) has elapsed from the operation time t, but in this embodiment, it predicts future control information ξp(t+Δ1), which is control information after only the outbound transmission delay time Δ1 has elapsed from the operation time t.

[0097] In the first embodiment, the transmission delay time measurement unit 24 sets the predicted time width Δ. In contrast, in the present embodiment, the transmission delay time measurement unit 24 sets the transmission delay time Δ1 instead of the predicted time width Δ. As a result, the remote machine control device 10 transmits, to the work machine device 80B, future control information ξp(t+Δ1), which is control information for the future, for only the outward transmission delay time.

[0098] The future operation information ξp(t+Δ1) transmitted from the remote machine operating device 10 arrives at the work machine 80 with a delay of time Δ1 when passing through the network 50. As a result, the working unit 83 of the work machine 80B performs a predetermined task based on the future operation information ξp(t) at the operation time t of the operation unit 11, i.e., with almost no outbound transmission delay.

[0099] Camera 84 captures the working status of working unit 83. Predictive video synthesis unit 86 synthesizes a predicted video of a future transmission delay time Δ2 with the monitoring video captured by camera 84, and supplies the synthesized predicted video as a monitoring video to video transmission unit 85. In other words, the monitoring video output from predicted video synthesis unit 86 is a video that has advanced into the future by transmission delay time Δ2.

[0100] The predicted image synthesis process by the predicted image synthesis unit 86 is a known technique and is disclosed, for example, in [B-7-20] "Low-delay and interactive zero-latency image-somatic integrated network (4) Delay compensation by image prediction in remote work." The image transmission unit 85 performs a predetermined image processing on the monitoring image synthesized with the predicted image, and transmits the obtained image-processed monitoring image to the remote machine control device 10.

[0101] The monitoring video transmitted from the work machine device 80B is delayed by a transmission delay time Δ2 when passing through the network 50, and arrives at the remote machine control device 10. The video receiving unit 13 of the remote machine control device 10 receives the monitoring video transmitted from the work machine device 80B. As a result, the monitor 14 displays the monitoring video with almost no transmission delay on the return path.

[0102] As described above, in the remote machine control system 1B according to this embodiment, the remote machine control device 10 predicts and transmits future operation information ξp(t+Δ1), which is operation information for the future by the outbound transmission delay time Δ1, to the work machine device 80B installed at a remote location. Meanwhile, the work machine device 80B synthesizes a predicted image for the future by the return transmission delay time Δ2 and transmits it as a monitoring image to the remote machine control device 10. This allows the operator to operate the work machine device 80B without feeling uncomfortable while checking the monitoring image of the work status of the work machine device 80B.

[0103] [Fourth embodiment] Next, a fourth embodiment will be described. Note that the same reference numerals are used to designate the same parts as those in the above-mentioned embodiments, and duplicated explanations will be omitted.

[0104] Fig. 14 is a block diagram showing the functional configuration of a remote machine control device and a work machine according to a fourth embodiment. The remote machine control system 1C includes a remote machine control device 10C instead of the remote machine control device 10 shown in Fig. 2. The remote machine control device 10C is obtained by adding a myoelectric potential information acquisition unit 15 that acquires myoelectric potential information from an operator and a myoelectric potential information processing unit 16 that performs predetermined signal processing on the acquired myoelectric potential information to the configuration of the remote machine control device 10 shown in Fig. 2.

[0105] The myoelectric potential information acquiring unit 15 is a plurality of myoelectric potential sensors attached to predetermined positions on the arm of an operator who operates the stick portion 11s of the operation unit 11. In this embodiment, the myoelectric potential information acquiring unit 15 is attached to six muscle bellies (two positions on the upper right upper arm and four positions on the forearm) of muscles used for movements such as elbow flexion and extension when operating the stick portion 11s. Specifically, the six muscle bellies are the biceps brachii, triceps brachii, palmaris longus, extensor carpi radialis longus, near the supinator, and flexor carpi radialis.

[0106] 15 is a waveform diagram showing an example of myoelectric potential information acquired by the myoelectric potential information acquisition unit 15. The six pieces of myoelectric potential information acquired from the six muscle belly locations described above appear as a result of muscle belly activation during movements such as elbow flexion, elbow extension, wrist flexion, exerting force on the wrist, wrist extension, wrist flexion, etc. However, the myoelectric potential information itself is weak and is greatly affected by surrounding electromagnetic noise and power supply hum noise, making it difficult to use it as is.

[0107] Therefore, the myoelectric potential information processing unit 16 converts the six pieces of myoelectric potential information acquired by the myoelectric potential information acquisition unit 15 into their respective muscle activity levels. The muscle activity level represents the mechanism from when muscle fibers are stimulated to when muscle tension is actually generated, based on a second-order lag model. Specifically, assuming that the muscle activity level is a and the absolute value of the myoelectric potential information is u, the relationship between the two is expressed by the formulas (1) and (2).

[0108]

number

[0109] Furthermore, since the myoelectric potential information and muscle activity of each muscle change drastically over time and are greatly affected by disturbances, it is problematic to use them directly to predict the control information. For this reason, in this embodiment, the myoelectric potential information processor 16 calculates the time pattern vector of muscle synergy.

[0110] A muscle synergy is a representation of a coordination structure observed when multiple muscles are activated simultaneously, using a time pattern (vector) consisting of multiple time-varying components, and a spatial pattern (matrix) consisting of a set of vectors that summarize the contribution of each component to the activity intensity of each muscle. Specifically, if the muscle synergy time pattern vector is H(t) and the spatial pattern matrix is ​​W, then Equation (3) holds.

[0111] A(t)=W H(t)+E(t) (3) where A(t) is the muscle activation vector and E(t) is the residual vector.

[0112] The number of components, i.e., the degree k of H(t), is set so that the values ​​of the elements of E(t) are sufficiently small compared to the values ​​of the corresponding elements of A(t). In this embodiment, k=3 is adopted as the smallest degree such that the values ​​of all elements of E(t) at any time t are less than 5% of the values ​​of the corresponding elements of A(t).

[0113] The six-dimensional muscle activity vector A(t) is obtained by converting six pieces of myoelectric potential information acquired from the arm when the stick unit 11s is operated, using equations (1) and (2). Furthermore, this muscle activity vector A(t) is expressed by equation (4) using a three-dimensional time pattern vector H(t) and a 6×3-dimensional space pattern matrix W.

[0114]

number

[0115] In formula (4), a1(t) to a6(t) on the left side are the elements of muscle activity vector A(t) calculated from myoelectric potential information at time t, and correspond to the muscle activity of the biceps brachii, triceps brachii, palmaris longus, extensor carpi radialis longus, supinator vicinity, and flexor carpi radialis at time t, respectively. Also, h1(t) to h3(t) on the right side are the elements of three-dimensional time pattern vector H(t) at time t, and hereafter, they are referred to as synergy 1, synergy 2, and synergy 3, respectively.

[0116] On the right-hand side, w11 to w63 and e1(t) to e6(t) are elements of the time-independent 6×3-dimensional spatial pattern matrix W and elements of the time-dependent residual vector E(t), respectively. Here, the 6×3-dimensional spatial pattern matrix W is extracted as a time-independent matrix having elements such that at any time t, each element of the residual vector E(t) is at its minimum value, using nonnegative matrix factorization (NMF).

[0117] FIG. 16 is a waveform diagram showing an example of each element of the extracted muscle synergy time pattern vector. Then, the future steering information prediction unit 25 shown in FIG. 14 predicts future steering information ξp(t+Δ), which is steering information after a first predetermined time has elapsed since the operation time t, by referring to, for the steering information ξ(t) at the operation time t, in addition to the past operating forces F (plurality of sampling data) during the period from the operation time t to a second predetermined time before the operation time t, the time patterns of past muscle synergies during the same period (the same number of sampling data).

[0118] (Learning process for future prediction model) In this embodiment, the machine learning device also performs machine learning processing of a future prediction model in the same manner as in steps S11 to S17 in Fig. 6. However, after step S11 in Fig. 6 is performed, the following processing is performed in steps S12 and S13.

[0119] In step S12, the machine learning device acquires, in addition to past operating forces (F(ta-w),...,F(ta)) that have been sampled and measured and normalized in the range from the operation time ta to a time before w, three synergies (h1(ta-w),...,h1(ta)), (h2(ta-w),...,h2(ta)), (h3(ta-w),...,h3(ta)) acquired in the same time range from Synergy 1, Synergy 2, and Synergy 3 that have been normalized in the range of -1.0 to 1.0 based on the maximum value of all of their absolute values.

[0120] In step S13, the machine learning device calculates (predicts) the future operation information ξp(ta+Δ) using a future prediction model that uses the operation information ξ(ta), past operation forces (F(ta-w), . . . , F(ta)), and three past synergies as inputs. After that, the process from step S14 onward shown in FIG. 6 is performed.

[0121] FIG. 17 is a diagram comparing predictions of control information (operational force) with and without myoelectric potential information. Here, the prediction width Δ=400 milliseconds. The diagram shows future control information (operational force) predicted using only the operation force for the control information (operational force) (Fx(t), Fy(t)) obtained by actual measurement, and future control information predicted using myoelectric potential information in addition to the operation force (both (Fxp(t) and Fyp(t)). Note that LSTM is again used as the machine learning model, and w=2.0 seconds is used.

[0122] In FIG. 17, the top row is a comparison of the operating force Fx, where the left side of the top row shows the result when prediction is made using only the operating force, and the right side shows the result when prediction is made using myoelectric potential information in addition to the operating force. The bottom row is a comparison of the operating force Fy, where the left side of the bottom row shows the result when prediction is made using only the operating force, and the right side shows the result when prediction is made using myoelectric potential information in addition to the operating force.

[0123] 17, it can be seen that the predicted future control information, Fpx(t) and Fpy(t), is closer to the actual measured value when myoelectric potential information is additionally used. In particular, in areas with large changes, there is a tendency that the control information change can be predicted earlier when myoelectric potential information is additionally used. This is because, for the right arm of the human body, the myoelectric potential change is detected about 50 milliseconds before the occurrence of muscle tension.

[0124] Although the case where the prediction width Δ=400 ms is shown here, it has been found that for shorter periods, such as up to Δ=200 ms, the actual measured value can be predicted almost correctly, regardless of whether or not EMG information is additionally used.

[0125] On the other hand, when predicting for a longer time, for example with a prediction width Δ of 600 ms ahead, the error increases, but by additionally using myoelectric potential information, it tends to be possible to predict the starting position of the change in the operating force more accurately. However, when the prediction width is Δ of 800 ms or more, the reproducibility deteriorates, and the results show that this method is not practical.

[0126] [Other embodiments] The present invention is not limited to the above-described embodiment, but is also applicable to, for example, remote control of automobiles or construction machines located in remote locations.

[0127] 1 and 2 has an operation unit 11 (FIG. 3) equipped with a stick unit 11s for operation in the above-mentioned embodiment, but may have a ring-shaped handle, for example, a steering handle used in an automobile, instead of the stick unit 11s. In this case, the steering handle has a neutral state, and when steered, a force is generated that tries to return to the neutral state, so that the present invention can be applied in the same manner as in the above-mentioned embodiment.

[0128] On the other hand, the work machine device 80 is applied as an automobile or construction machine itself located in a remote location. In this case, the working unit 83 corresponds to the steering mechanism of an automobile or the like, and the camera 84 captures images of the front direction of the automobile, etc. As a result, the monitor 14 of the remote machine control device 10 displays an image of the scenery that the driver can see through the windshield of the automobile or the like. The operator can then remotely control the automobile or the like located in a remote location with the feeling of actually driving the automobile or the like. [Explanation of symbols]

[0129] 1,1B,1C Remote Machine Control System 10 10C Remote machine control device 25, 25A Future flight information prediction section 50 Network 80,80B Work machinery equipment

Claims

1. A remote machine control device for an operator to operate a control unit while viewing a monitoring image to control a work machine at a remote location, an operation force acquisition unit that acquires an operation force applied to the operation unit; a future prediction model storage unit that stores a future prediction model, which is a machine learning model trained using, as teacher data, operation information at the time of operation of the operation unit, operation information at a time after a transmission delay time between the operation unit and the work machine device from the operation time, and time-series data of the operation force within a predetermined range in the past from the operation time; a future operation information prediction unit that predicts future operation information, which is future operation information after a time equivalent to the transmission delay time between the work machine device and the operation unit, based on the operation information obtained from the operation unit operated by the operator, the operation force acquired by the operation force acquisition unit, and the future prediction model held in the future prediction model holding unit, with the operation time of the operation unit as a reference; a transmission unit that transmits the future steering information predicted by the future steering information prediction unit to the work machine device; A remote machine control device equipped with a

2. a myoelectric potential information acquisition unit for acquiring arm myoelectric potential information related to the operation of the operator, the future prediction model is the machine learning model trained further using time-series data of the myoelectric potential information, The future operation information prediction unit predicts the future operation information by further using the myoelectric potential information acquired by the myoelectric potential information acquisition unit.

2. The remote machine control system of claim 1.

3. A remote machine control device for allowing an operator to operate a control unit while watching a monitoring image to control a work machine at a remote location, a transmission delay time measuring unit that measures a transmission delay time between the operation unit and the work machine device based on an operation time in the operation unit; a transmission delay time adjustment dial for continuously adjusting the transmission delay time measured by the transmission delay time measurement unit in response to an operation by the operator; a future operation information prediction unit that predicts future operation information, which is future operation information after a time equivalent to the transmission delay time adjusted by the transmission delay time adjustment dial has elapsed, based on operation information obtained from the operation unit operated by the operator, predetermined information related to the operation of the operator, and a future prediction model, with the operation time of the operation unit as a reference; a transmission unit that transmits the future steering information predicted by the future steering information prediction unit to the work machine device; A remote machine control device equipped with a

4. A program for causing a computer to function as each part of the remote machine control device according to claim 1.

5. A remote machine control device according to claim 1 or 3; The work machine device receives the future operation information transmitted from the remote machine control device and transmits the monitoring image to the remote machine control device; A remote machine control system comprising:

Images