Remote machine control device, recording medium, program, and remote machine control system

Abstract

A remote machine control device according to one aspect of the present invention enables an operator to operate a work machine device in a remote location by operating an operation unit while viewing a monitoring video, and comprises: an operation force acquisition unit that acquires an operation force for the operation unit; a future prediction model storage unit that stores a future prediction model that is a machine learning model trained by using, as teacher data, control information at the operation time of the operation unit, control information at a time coming a transmission delay to the work machine device after the operation time, and time series data of the operation force within a predetermined range before the operation time; a prediction unit that, on the basis of control information obtained from the operation unit operated by the operator, the operation force acquired by the operation force acquisition unit, and the future prediction model stored in the future prediction model storage unit, predicts future control information which is control information in the future following a passage of time corresponding to the transmission delay to the work machine device with respect to the operation time for the operation unit; and a transmission unit that transmits the future control information predicted by the prediction unit to the work machine device.

Description

Remote machine control device, recording medium, program, and remote machine control system

[0001] The present invention relates to a remote machine control device, a recording medium, a program, and a remote machine control system.

[0002] Conventionally, remote machine control systems have been known in which an operator remotely controls a machine installed in a remote location via a network. Specifically, the operator controls the machine by operating a control lever at hand while monitoring video of the machine's movement. In such remote machine control systems, there is a problem in that the operator feels a discrepancy between the monitoring video and the actual lever operation due to network transmission delays.

[0003] Therefore, a technology has been disclosed that reduces the impact of communication delays between the operation terminal (operator's operating lever) and the actual robot (machine) on the operator's operation (see Patent Document 1).

[0004] The technology of Patent Document 1 involves monitoring video footage of the movements of an actual robot operating in a remote location with a delay due to communication delays by modeling the actual robot and its surrounding objects, synthesizing predicted video that compensates for communication delays based on these models, and displaying this as monitoring video.

[0005] Japanese Patent Application Laid-Open No. 2020-26028

[0006] In the technology of Patent Document 1, the displayed monitoring image is highly dependent on the accuracy of the model that is the basis for the predictive synthesis. However, since the objects around the actual robot are not necessarily specific targets, it is difficult to always accurately model the objects around the robot, which results in the problem that an appropriate monitoring image cannot be displayed.

[0007] The present invention has been proposed in consideration of the above-mentioned circumstances, and provides a remote machine control device, a recording medium, a program, and a remote machine control system that enable a user to operate a machine installed in a remote location without feeling uncomfortable while monitoring the machine's movement video.

[0008] a future prediction model holding unit that holds a future prediction model, which is a machine learning model trained using, as teacher data, operation information at the time of operation of the operation unit, operation information at a time after a transmission delay time between the operation time and the work machine, and time series data of the operation force within a predetermined range in the past from the operation time; a prediction unit that predicts future operation information, which is future operation information after a time equivalent to the transmission delay time between the operation unit and the work machine, based on the operation information obtained from the operation unit operated by the operator, the operation force acquired by the operation force acquisition unit, and the future prediction model held in the future prediction model holding unit, and a transmission unit that transmits the future operation information predicted by the prediction unit to the work machine.

[0009] A remote machine control system that is one aspect of the present invention is a remote machine control device for an operator to operate a control unit while viewing monitoring video to control a work machine at a remote location, and is equipped with a transmission delay time measurement unit that measures a transmission delay time between the work machine and the operation unit based on the operation time of the operation unit; a transmission delay time adjustment dial that continuously adjusts the transmission delay time measured by the transmission delay time measurement unit in accordance with the operation of the operator; a prediction unit that predicts future operation information, which is future operation information after a time equivalent to the transmission delay time adjusted by the transmission delay time adjustment dial, based on the operation time of the operation unit, based on the operation information obtained from the operation unit operated by the operator, specified information related to the operation of the operator, and a future prediction model; and a transmission unit that transmits the future operation information predicted by the prediction unit to the work machine.

[0010] A recording medium according to one aspect of the present invention stores a program for causing a computer to function as each part of the remote machine control device.

[0011] A program according to one aspect of the present invention causes a computer to function as each part of the remote machine control device.

[0012] A remote machine control system according to one aspect of the present invention comprises the remote machine control device and the work machine device that receives the future control information transmitted from the remote machine control device and transmits the monitoring video to the remote machine control device.

[0013] The present invention reduces the effects of transmission delays between machines installed in remote locations, allowing operators to operate the machines without feeling uncomfortable while viewing the monitoring video.

[0014] 1 is a schematic diagram of a remote machine control system according to a first embodiment. FIG. 1 is a block diagram showing the functional configuration of a remote machine control device and a work machine. FIG. 2 is a schematic diagram showing an example of an operation unit. FIG. 3 is a block diagram showing the functional configuration of a future operation information prediction unit. FIG. 4 is a flowchart showing a prediction processing routine by the operation information prediction unit. FIG. 5 is a flowchart showing machine learning processing of a future prediction model. FIG. 6 is a waveform diagram showing an example of operation force (operation information) obtained from the operation unit. FIG. 7 is a schematic diagram showing a comparison between operation information (operation force) and future operation information. FIG. 8 is a comparison diagram between actual operation information (operation force) and future operation information. FIG. 9 is a block diagram showing the functional configuration of a future operation information prediction unit according to a second embodiment. FIG. 10 is a flowchart showing a prediction processing routine by the operation information prediction unit according to the second embodiment. FIG. 11 is a flowchart showing a model update routine which is a subroutine of step S7. FIG. 12 is a block diagram showing the functional configuration of a remote machine control device and a work machine according to a third embodiment. FIG. 13 is a block diagram showing the functional configuration of a remote machine control device and a work machine according to a fourth embodiment. FIG. 14 is a waveform diagram showing an example of myoelectric potential information acquired by an electromyographic information acquisition unit. FIG. 15 is a waveform diagram showing an example of each element of a time pattern vector of extracted muscle synergies. FIG. 16 is a comparison diagram between actual operation information and future operation information depending on the presence or absence of electromyographic potential information.

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0016] 1 is a schematic diagram of a remote machine control system 1 according to a first embodiment. The remote machine control system 1 includes a remote machine control device 10 that is operated by an operator, and a work machine device 80 that performs a predetermined task while communicating with the remote machine control device 10 via a network 50.

[0017] The work machine device 80 is located at a location, for example, a remote location, away from the remote machine control device 10. The work machine device 80 may be fixed to the location where it is located, or may be movable from the location where it is located.

[0018] The camera 84 of the work machine device 80 is installed in a position where it can capture an image of the working unit 83 performing a predetermined task. The camera 84 captures images of the work status of the working unit 83 and generates monitoring images. The work machine device 80 transmits the generated monitoring images to the remote machine control device 10 via the network 50.

[0019] The monitor 14 of the remote machine control device 10 displays the monitoring video transmitted from the work machine 80, i.e., the work status of the working unit 83. The operator operates the operation unit 11 of the remote machine control device 10 while viewing the monitoring video displayed on the monitor 14, to operate the work machine 80 (including the working unit 83 included therein) located in a remote location. In this embodiment, the operator's movement of the operation unit 11 is referred to as "operation," and the operator's movement of the work machine 80 (including the working unit 83) is referred to as "operation."

[0020] In this way, when the operator operates the operation unit 11, the remote machine control device 10 transmits control information to the work machine device 80. Then, when the work machine device 80 receives the control information, it performs an operation corresponding to the operation of the operation unit 11.

[0021] When information is transmitted between the remote machine control device 10 and the work machine 80, a transmission delay occurs due to the network 50. The transmission delay time associated with the information transmission from the remote machine control device 10 to the work machine 80 (outbound) is set to Δ1.

[0022] When the remote machine control device 10 transmits control information at time t1 and the work machine device 80 receives the control information at time t2, the following equation holds: t2=t1+Δ1

[0023] Meanwhile, the transmission delay time associated with the information transmission from the work machine device 80 to the remote machine control device 10 (return trip) is assumed to be Δ2. When the work machine device 80 transmits monitoring video of the work status of the working unit 83 captured by the camera 84 at time t'2, and the remote machine control device 10 receives and displays that monitoring video at time t3, the following equation holds: t3 = t'2 + Δ2

[0024] The return path transmission delay time Δ2 includes not only the delay caused by the information transmission over the network 50, but also the delay caused by the image processing of the monitoring video performed before and after the information transmission. As a result, the transmission delay time Δ1 and the transmission delay time Δ2 do not necessarily coincide.

[0025] Here, assuming that t2 = t'2, t3 = t1 + Δ1 + Δ2. In other words, when the operator operates the operation unit 11 of the remote machine control device 10 at time t1, the monitor 14 displays the working status of the working unit 83 resulting from that operation at time t3.

[0026] 2 is a block diagram showing the functional configuration of the remote machine control device 10 and the work machine device 80. The remote machine control device 10 includes an operation unit 11 operated by a user, a control information prediction unit 12 that predicts future control information, a video receiving unit 13 that receives monitoring video, and a monitor 14 that displays the work status of the work machine device 80 as monitoring video.

[0027] 3 is a schematic diagram showing an example of the operation unit 11. The operation unit 11 is not particularly limited as long as it is operated by an operator, but in this embodiment it is an operation lever. The operation unit 11 includes a stick unit 11s for operating the work machine device 80 and a dial adjustment unit 11d for manually adjusting a predicted time width Δ, which will be described later. The predicted time width Δ adjusted by the dial adjustment unit 11d is used by the operation information prediction unit 12.

[0028] The stick 11s is rod-shaped and stands upright perpendicular to the base when in a neutral state. The stick 11s can be tilted in any direction within the xy plane in response to an operator's operation. The angle at which the stick 11s is tilted (displaced) in any direction from the neutral state is called the displacement angle θ, and the projections of this angle onto the x-axis and y-axis, i.e., the x-component and y-component, are called the displacement angle θx and the displacement angle θy, respectively.

[0029] Spring-like members that can expand and contract in the x and y directions are attached to the base of the stick 11s. When the operator tilts the stick 11s from its neutral position by a displacement angle θ, the spring-like members apply forces to the stick 11s that are simply proportional to the displacement angles θx and θy, respectively, to return the stick 11s to the neutral position. The stick 11s is also provided with sensors that output signals corresponding to the expansion of the respective spring-like members.

[0030] When the stick unit 11s is tilted by a displacement angle θ by operation of the operator, the operation unit 11 supplies a signal corresponding to the displacement angle θ from the neutral state and signals corresponding to the extension of each spring-like member in the x and y directions to the operation information prediction unit 12. Hereinafter, any time when the operator continuously operates the operation unit 11 is defined as operation time t.

[0031] As shown in FIG. 2, the operation information prediction unit 12 includes a displacement information acquisition unit 21, a calculation unit 22, an operation force acquisition unit 23, a transmission delay time measurement unit 24, a future operation information prediction unit 25, and a transmission unit 26.

[0032] The displacement information acquisition unit 21 receives a signal corresponding to the displacement angle θ from the neutral state of the operation unit 11, and detects the displacement angle θ for each sampling period. The displacement information acquisition unit 21 supplies the detected displacement angle θ to the calculation unit 22.

[0033] The calculation unit 22 calculates steering information ξ(θ) for each sampling period using the displacement angle θ supplied from the displacement information acquisition unit 21 as an input, and supplies the steering information ξ(θ) to the future steering information prediction unit 25. The steering information ξ(θ) is information for steering the work machine device 80, and is expressed as a predetermined function of the displacement angle θ.

[0034] The operating force acquisition unit 23 receives signals corresponding to the extensions in the x and y directions of the spring-like member provided in the operating unit 11, acquires the operating force F applied to the operating unit 11 by the operator for each sampling period, and supplies this to the future operation information prediction unit 25. Hereinafter, the operating force applied in the x direction is designated as Fx, and the operating force applied in the y direction is designated as Fy.

[0035] The transmission delay time measurement unit 24 measures a predicted time width Δ when predicting future steering information. The predicted time width Δ is the sum of the transmission delay time Δ1 associated with the information transmission on the outbound route and the transmission delay time Δ2 associated with the information transmission on the return route.

[0036] The method for measuring the predicted time width Δ is not particularly limited as long as it can be measured by the transmission delay time measurement unit 24. The transmission delay time measurement unit 24 may provide a transmission delay time Δmeasure (fixed value) measured in advance in a preparation stage as the predicted time width Δ. Alternatively, the predicted time width Δ may be measured using an "echo request / echo reply" message of the Internet Control Message Protocol (ICMP), which is used in a Ping command that checks the connection status on a network.

[0037] Furthermore, as described above, the transmission delay time Δ1 and the transmission delay time Δ2 do not necessarily coincide. Therefore, if the internal clocks of the remote machine control device 10 and the work machine device 80 are synchronized with high precision (for example, on the order of sub-milliseconds) by sharing GPS information, for example, the following method is possible. That is, the transmission delay time measurement unit 24 may calculate the transmission delay times Δ1 and Δ2 using the transmission time and reception time (of information) recorded by the remote machine control device 10 and the work machine device 80, respectively, and then measure the predicted time width Δ by taking the sum of these values.

[0038] However, if the predicted time width Δ measured as described above is used as is to predict the operation information, the monitoring image may show the work machine device 80 (for example, the working unit 83) reacting excessively to the operator's operation, which may cause the operator to feel uncomfortable. To prevent this kind of discomfort, when the operator operates the adjustment dial 11d of the operation unit 11 shown in Figure 3, the transmission delay time measurement unit 24 may adjust the size of the predicted time width Δ in accordance with the amount of operation.

[0039] The future control information prediction unit 25 predicts, for each sampling period, future control information ξp(t+Δ), which is control information after a first predetermined time has elapsed since the operation time t, based on the control information ξ(t) as a function of the displacement angle θ(t) at the operation time t. Note that the first predetermined time is equal to the predicted time width Δ measured by the transmission delay time measurement unit 24.

[0040] Specifically, the future operation information prediction unit 25 predicts future operation information ξp(t+Δ), which is the operation information after a first predetermined time has elapsed from the operation time t, by referring to the past operation force F (multiple sampling data) within the period from the operation time t to the second predetermined time before the operation time t, for the operation information ξ(t) at the operation time t.

[0041] The reason why the future control information prediction unit 25 can predict the future control information ξp(t+Δ) using the sampling data of the past operation force F is as follows. As described above, the control information ξ(t) is a function of the displacement angle θ(t) of the operation unit 11. The displacement angle θ(t) is determined by the balance between the force applied by the operator to the operation unit 11 and the force caused by the spring-like member that the operator feels as a response from the operation unit 11. For this reason, the future control information prediction unit 25 can predict the future control information ξp(t+Δ) with high accuracy by using the response of the operation unit 11 up to that point, that is, the sampling data of the past operation force F. Specifically, the future control information prediction unit 25 is configured as shown in FIG. 4 .

[0042] 4 is a block diagram showing the functional configuration of the future operation information prediction unit 25. The future operation information prediction unit 25 includes a future prediction model holding unit 31 that holds a future prediction model, a prediction unit 32 that predicts future operation information ξp(t+Δ), and a operation information holding unit 33 that holds operation information ξ(t) at operation time t and future operation information ξp(t+Δ).

[0043] The future prediction model held in the future prediction model holding unit 31 is, for example, a machine learning model that has been externally trained. The future prediction model has been trained by machine learning using a large amount of teacher data, and thereby establishes a relationship between the operation information ξ(t) at operation time t, the past operation force F (plurality of sampling data) within a period from the operation time t to a second predetermined time before, and the future operation information ξp(t+Δ).

[0044] The prediction unit 32 predicts future maneuver information ξp(t+Δ) using the future prediction model held in the future prediction model holding unit 31. Specifically, the prediction unit 32 inputs, into the future prediction model, the maneuver information ξ(t) at operation time t supplied from the calculation unit 22 shown in FIG. 2 and the past operation force F=(Fy, Fy) (plurality of sampling data within a period from the operation time t to a second predetermined time before) supplied from the operation force acquisition unit 23. Then, the prediction unit 32 outputs information output from the future prediction model as future maneuver information ξp(t+Δ).

[0045] The operation information holding unit 33 holds the operation information ξ(t) at the operation time t supplied from the calculation unit 22 and the future operation information ξp(t) predicted by the prediction unit 32 in association with each other.

[0046] 2 , the future steering information prediction unit 25 supplies the predicted future steering information ξp(t+Δ) to the transmission unit 26. The transmission unit 26 transmits the future steering information ξp(t+Δ) predicted by the future steering information prediction unit 25 to the work machine device 80 via the network 50.

[0047] In this way, the remote machine control device 10 transmits the future control information ξp(t+Δ) at the operation time t to the work machine device 80. Here, the future control information ξp(t+Δ) refers to control information that is predicted to be generated by the operator in the future after the predicted time width Δ (=Δ1+Δ2) has elapsed from the operation time t.

[0048] The future manipulation information ξp(t+Δ) (=ξp(t+Δ1+Δ2)) transmitted from the remote machine manipulation device 10 is delayed by time Δ1 when passing through the network 50. For this reason, the future manipulation information received by the work machine device 80 at the operation time (t+Δ1) is ξp(t+Δ2).

[0049] As shown in Figure 2, the work machine device 80 is equipped with a receiving unit 81 that receives future operation information ξp(t+Δ) from the remote machine operation device 10, a control unit 82 that generates machine control information from the received operation information, a working unit 83 that performs specified work in accordance with the machine control information generated by the control unit 82, a camera 84 that captures images of the work status of the working unit 83 and generates monitoring images, and an image transmitting unit 85 that transmits the monitoring images of the working unit 83 obtained by the camera 84.

[0050] The control unit 82 controls the working unit 83 using machine control information generated based on the future operation information ξp(t+Δ) received by the receiving unit 81. As a result, the working unit 83 performs a predetermined task based on the future operation information ξp(t+Δ2) after the time Δ1 has elapsed since the operation time t. The camera 84 captures images of the working status of the working unit 83. The video transmission unit 85 performs predetermined image processing, such as video compression processing, on the monitoring video of the working unit 83 captured by the camera 84, and transmits the obtained image-processed monitoring video to the remote machine control device 10.

[0051] The monitoring video transmitted from the work machine device 80 is delayed by time Δ2 when it travels through the network 50 and arrives at the remote machine control device 10. The video receiving unit 13 of the remote machine control device 10 then receives the monitoring video transmitted from the work machine device 80, performs predetermined image processing on the video, and supplies the processed video to the monitor 14. The monitor 14 displays the monitoring video supplied from the video receiving unit 13.

[0052] The future operation information ξp(t+Δ1+Δ2) transmitted from the remote machine control device 10 arrives at the work machine device 80 with a delay of time Δ1 as it travels through the network 50. As a result, the working unit 83 of the work machine device 80 performs a predetermined task based on the future operation information ξp(t+Δ2) with respect to the operation time t of the operation unit 11, i.e., a task that is a time Δ2 in the future from the operation time t. Then, the monitoring video of the working unit 83 captured by the camera 84 is displayed on the monitor 14 of the remote machine control device 10 with a delay of time Δ2 as it travels through the network 50.

[0053] As a result, the monitor 14 displays a monitoring image of the predetermined task based on the future operation information ξp(t) at the operation time t of the operation unit 11, i.e., the task status that was predicted to occur at the same time as the operation time t. Therefore, the operator can operate the operation unit 11 without being affected by the transmission delay of the network 50, in other words, without feeling any discomfort from the monitoring image displayed on the monitor 14.

[0054] 5 is a flowchart showing a prediction processing routine performed by the operation information prediction unit 12. In step S1, the displacement information acquisition unit 21 detects whether or not the operation unit 11 has been operated. If the operation unit 11 has been operated, the process proceeds to the next step.

[0055] In step S2, the displacement information acquisition unit 21 acquires the displacement angle θ(t) of the operation unit 11 at the operation time t. The calculation unit 22 generates operation information ξ(t) based on a predetermined function that uses the displacement angle θ(t) acquired by the displacement information acquisition unit 21 as an input, and supplies the obtained operation information ξ(t) to the future operation information prediction unit 25.

[0056] The operating force acquisition unit 23 acquires the operating force applied to the operating unit 11 through a sensor provided on a spring-like member at the base of the stick portion 11s of the operating unit 11, and supplies the operating force to the future driving information prediction unit 25.

[0057] In step S3, the transmission delay time measurement unit 24 measures a predicted time width Δ corresponding to the transmission delay time caused by the round-trip information transmission between the remote machine control device 10 and the work machine device 80. The predicted time width Δ may be an actual measurement value measured by the above-mentioned conventional technology, or may be a preset fixed value.

[0058] In step S4, the future operation information prediction unit 25 predicts future operation information ξp(t+Δ), which is the operation information after the first predetermined time has elapsed since the operation time t, by referring to the past operation force F (plurality of sampling data) within the period from the operation time t to the second predetermined time before the operation time t, for the operation information ξ(t) at the operation time t supplied from the calculation unit 22.

[0059] In step S5, the transmission unit transmits the future steering information ξp(t+Δ) predicted by the future steering information prediction unit 25 to the work machine device 80.

[0060] In step S6, it is determined whether or not the operation of the work machine device 80 has ended, and if it is determined that the operation has not ended, the process returns to step S1. If it is determined that the operation of the work machine device 80 has ended, the series of processes ends.

[0061] Next, the machine learning process of the future prediction model will be described. The future prediction model is a machine learning model that is trained using the operation information ξ(t) and the operation force F(t) at the operation time t that are obtained in advance by actual measurement as training data.

[0062] 6 is a flowchart showing the machine learning process of the future prediction model. That is, the machine learning device (not shown) performs the machine learning process of the future prediction model according to the steps from step S11 onwards.

[0063] In step S11, the machine learning device sets an initial value w at the operation time ta and acquires the operation information ξ(ta), where w is the second predetermined time mentioned above.

[0064] In step S12, the machine learning device acquires past operating forces (F(ta-w), . . . , F(ta)) sampled and measured in the range from the operation time ta to the time before w.

[0065] In step S13, the machine learning device calculates (predicts) future operation information ξp(ta+Δ), which is operation information at an operation time (ta+Δ) that is a first predetermined time Δ after the operation time ta, using a future prediction model that inputs the operation information ξ(ta) and past operation forces (F(ta-w), ..., F(ta)).

[0066] In step S14, the machine learning device acquires operation information ξ(ta+Δ) which is an actual measurement value at the operation time (ta+Δ).

[0067] In step S15, the machine learning device updates the internal parameters of the future prediction model in a direction that reduces the difference between the future driving information ξp(ta+Δ) and the corresponding driving information ξ(ta+Δ), which is an actual measured value as training data.

[0068] In step S16, the machine learning device increases the operation time ta by a time δ, where δ is the sampling time interval.

[0069] In step S17, the machine learning device ends the process if the actual measurement value ξ(ta) of the operation information at the operation time ta does not exist, and returns to step S12 if the actual measurement value ξ(ta) of the operation information does exist.

[0070] As described above, in the remote machine control system 1 according to this embodiment, the remote machine control device 10 predicts and transmits future control information, which is control information for the future, by an amount corresponding to the round-trip transmission delay time, to the work machine device 80 installed in a remote location. This allows the operator to operate the work machine device 80 without feeling uncomfortable while checking the monitoring video of the work status of the work machine device 80.

[0071] (Measurement Example) Next, a specific example will be given to explain the calculation (prediction) of future control information ξp(t+Δ) using the actual measurement value of the control information. Although the control information ξ was stated to be expressed as a predetermined function of the displacement angle θ, in the following, to reduce the calculation load, it is assumed to be simply proportional to the displacement angle θ, that is, a linear function of the displacement angle θ. On the other hand, as mentioned above, the operating force F is also simply proportional to the displacement angle θ. Therefore, when a constant k is used, the following relationship holds between the control information ξ and the operating force F: ξ = k F

[0072] In this embodiment, k = 1 is set to further simplify the calculation. That is, the operation force F is also used as the control information ξ in its original form. Therefore, the control information ξ(t) at time t is equivalent to the operation force F(t), and its x-component and y-component are also expressed as Fx(t) and Fy(t), respectively. The x-component and y-component of future control information ξp(t+Δ), which is control information for a predicted time width Δ after time t, are also expressed as Fpx(t+Δ) and Fpy(t+Δ), respectively. Here, it is assumed that the x-component and y-component of the control information ξ (operation force F) can be handled independently of each other.

[0073] 7 is a waveform diagram showing an example of the operating force F (steering information ξ) obtained from the operating unit 11. The vertical axis indicates the x-component (Fx) and y-component (Fy) of the operating force F output in sample units from the operating unit 11, normalized to a range of −1.0 to 1.0 based on the maximum absolute value of those components. The horizontal axis indicates the time axis (seconds). Each thick solid line plots the first 1000 samples (20 seconds worth) sampled at a sampling frequency of 50 Hz.

[0074] The following mainly describes how to handle Fx, which is the x component of the maneuver information ξ, but Fy, which is the y component, can also be handled in the same way.

[0075] 8 is a diagram schematically illustrating a comparison between the operation information ξ (operation force Fx) obtained by actual measurement and the future operation information ξp (future operation force Fpx) predicted using the same. The future operation information ξp(t+Δ) (future operation force Fpx(t+Δ)) is predicted from the operation information ξ(ta) (operation force Fx(ta)) at the operation time ta and sample data of the past operation force F (Fx(ta-w), . . . , Fx(ta)) during the period from the operation time ta to the time w before. Note that in this embodiment, since the operation information ξ is equivalent to the operation force Fx, Fx(ta) is not only the operation information ξ(ta) actually measured at the operation time ta, but also one of the sample data of the past operation force F used to predict ξp(t+Δ).

[0076] 9 is a diagram comparing the actual operation information ξ (operating force F) with the future operation information ξ (future operating force Fp) obtained through an experiment. Here, the prediction time width Δ, which is the first predetermined time described above, is set to 500 milliseconds, and the actually measured values ​​of the operating forces Fx(t) and Fy(t) are compared with the future operating forces Fpx(t) and Fpy(t) predicted using the future prediction model.

[0077] In the experiment shown in FIG. 9, a machine learning model that has undergone the machine learning process shown in FIG. 6, specifically, a Long Short Term Memory (LSTM), was used as the future prediction model.

[0078] Furthermore, the acquisition range period w for the past operating force sampling data (Fx(t-w), ..., Fx(t)) and (Fy(t-w), ..., Fy(t)), which corresponds to the second predetermined period described above, was set to w = 2.0 seconds.

[0079] Furthermore, the sampling frequency of the actual maneuver information ξ (operating force F) was set to 50 Hz. Therefore, the past operating force F required to predict the future maneuver information ξ (future operating force F) was sampled at a time width of 20 milliseconds within the acquisition range period of 2 seconds, resulting in 100 samples being obtained as the input past operating force F.

[0080] This indicates that, starting from a certain operation time t, the future operation information ξp 0.5 seconds after the operation time t is predicted using the operation force F sampled for the past 2.0 seconds. Therefore, as shown by the dotted line in Fig. 9, for example, the future operation force Fpx(t) first appears at the operation time t = 2.5 seconds.

[0081] In this embodiment, the LSTM is used as an example of the machine learning model, but the machine learning model is not particularly limited as long as it predicts time-series data. For example, the machine learning model may be a GRU (Gated Recurrent Unit), which is a recurrent neural network, or a time convolution network.

[0082] In this embodiment, the x component and the y component of the steering information ξ (operating force F) are not dependent on each other and can be handled independently.

[0083] However, when an interaction between the x and y components of the steering information ξ (operating force F) is expected, machine learning processing may be performed on the machine learning model for calculating Fpx(t+Δ) using not only the x components (Fx(t−w), ..., Fx(t)) of past operating force vectors but also their y components (Fy(t−w), ..., Fy(t)) as training data.

[0084] 4 stores the steering information ξ(t) at the operation time t and the future steering information ξp(t) predicted by the prediction unit 32 in association with the operation time t. Therefore, the information held in the steering information holding unit 33 may be retrieved to the outside of the system at a predetermined timing, and the retrieved information may be used as teacher data or the like to perform machine learning processing and update the internal parameters of the future prediction model. In this case, the future prediction model held in the future prediction model holding unit 31 may be replaced with the future prediction model updated as described above under predetermined conditions.

[0085] [Second Embodiment] Next, a second embodiment will be described. The same components as those in the first embodiment are denoted by the same reference numerals, and redundant descriptions will be omitted. In the second embodiment, a future operation information prediction unit 25A shown in FIG. 10 is used instead of the future operation information prediction unit 25 shown in FIG. 4.

[0086] Fig. 10 is a block diagram showing the functional configuration of a future steering information prediction unit 25A according to the second embodiment. The future steering information prediction unit 25A is obtained by adding a future prediction model update unit 34 to the configuration of the future steering information prediction unit 25 shown in Fig. 4. The future prediction model update unit 34 compares the steering information ξ(t) at operation time t supplied from the calculation unit 22 with the future steering information ξp(t) predicted at the operation time (t-Δ) and held in the steering information holding unit 33, and updates the future prediction model held in the future prediction model holding unit 31.

[0087] 11 is a flowchart showing a prediction processing routine by the operation information prediction unit 12A (not shown) including the future operation information prediction unit 25A. Here, the prediction processing routine executes steps S1 to S5 as in FIG. 5, then executes a new step S7, and finally executes step S6.

[0088] In step S7, the future prediction model update unit 34 compares the operation information ξ(t) with the future operation information ξp(t) and updates the future prediction model held in the future prediction model holding unit 31. Specifically, the model update routine shown in FIG. 12 is executed.

[0089] Fig. 12 is a flowchart showing a model update routine, which is a subroutine of step S7 shown in Fig. 11. In step S21, the future prediction model update unit 34 acquires the operation information ξ(t) at the operation time t supplied from the calculation unit 22.

[0090] In step S22, the future prediction model update unit 34 acquires future operation information ξp(t) at operation time t, which is predicted at operation time (t-Δ), which is a prediction time width Δ in the past from the operation time t, from the future operation information predicted in the past and stored in the operation information storage unit 33.

[0091] In step S23, the future prediction model update unit 34 updates the internal parameters of the future prediction model stored in the future prediction model storage unit 31 so as to reduce the difference between the operation information ξ(t) and the future operation information ξp(t). Then, the future prediction model update unit 34 repeatedly executes steps S21 to S23 each time the operation time t advances.

[0092] However, if the future prediction model update unit 34 were to update the future prediction model every time sample data of the steering information ξ(t) and the future steering information ξp(t) is supplied, large-scale and high-speed computing power would be required, resulting in a large computational load. Therefore, the future prediction model update unit 34 may update the future prediction model for each sample data of the steering information ξ(t) and the future steering information ξp(t) every predetermined number of samples (e.g., 20 samples).

[0093] As described above, the remote machine control device 10 according to this embodiment, in parallel with predicting future operation information, updates the future prediction model as needed so that the previously predicted future operation information matches the actual operation information. Therefore, even if a change occurs in the work situation of the work machine device 80 and the prediction accuracy of the future operation information decreases, the discomfort felt by the operator who is operating the work machine device 80 while checking the monitoring video of the work situation can be gradually reduced.

[0094] Third Embodiment Next, a third embodiment will be described. Note that the same components as those in the above-described embodiment are denoted by the same reference numerals, and redundant description will be omitted.

[0095] Figure 13 is a block diagram showing the functional configuration of a remote machine control device and a work machine according to the third embodiment. The remote machine control system 1B is equipped with a work machine 80B instead of the work machine 80 shown in Figure 2. The work machine 80B has the same configuration as the work machine 80 shown in Figure 2, but with the addition of a predicted image synthesis unit 86 that synthesizes predicted images.

[0096] In the first embodiment, the remote machine control device 10 predicts future control information ξp(t+Δ), which is control information after a predicted time width Δ (=Δ1+Δ2) has elapsed from the operation time t, but in this embodiment, it predicts future control information ξp(t+Δ1), which is control information after only the outbound transmission delay time Δ1 has elapsed from the operation time t.

[0097] In the first embodiment, the transmission delay time measurement unit 24 sets the predicted time width Δ. In contrast to this, in the present embodiment, the transmission delay time measurement unit 24 sets a transmission delay time Δ1 instead of the predicted time width Δ. As a result, the remote machine control device 10 transmits, to the work machine device 80B, future control information ξp(t+Δ1), which is control information in the future for only the outbound transmission delay time.

[0098] The future operation information ξp(t+Δ1) transmitted from the remote machine operating device 10 is delayed by time Δ1 when it passes through the network 50 and arrives at the work machine device 80. As a result, the working unit 83 of the work machine device 80B performs a predetermined task based on the future operation information ξp(t) at the operation time t of the operating unit 11, i.e., with almost no outbound transmission delay.

[0099] Camera 84 captures images of the work status of working unit 83. Predicted image synthesis unit 86 synthesizes predicted images of a future transmission delay time Δ2 with the monitoring images captured by camera 84, and supplies the synthesized predicted images as monitoring images to image transmission unit 85. In other words, the monitoring images output from predicted image synthesis unit 86 are images that have advanced into the future by transmission delay time Δ2.

[0100] The predicted video synthesis process by the predicted video synthesis unit 86 is a known technique and is disclosed, for example, in [B-7-20] "Low-delay, interactive, zero-latency video / Somatic integrated network (4) Delay compensation by video prediction in remote work." The video transmission unit 85 performs predetermined image processing on the monitoring video synthesized with the predicted video, and transmits the obtained image-processed monitoring video to the remote machine control device 10.

[0101] The monitoring video transmitted from the work machine device 80B is delayed by a transmission delay time Δ2 when passing through the network 50, and arrives at the remote machine control device 10. The video receiving unit 13 of the remote machine control device 10 receives the monitoring video transmitted from the work machine device 80B. As a result, the monitor 14 displays the monitoring video with almost no transmission delay on the return path.

[0102] As described above, in the remote machine control system 1B according to this embodiment, the remote machine control device 10 predicts and transmits future control information ξp(t+Δ1), which is control information for the future by the outbound transmission delay time Δ1, to the work machine 80B installed in a remote location. Meanwhile, the work machine 80B synthesizes a predicted image for the future by the inbound transmission delay time Δ2 and transmits this as a monitoring image to the remote machine control device 10. This allows the operator to operate the work machine 80B without feeling uncomfortable while checking the monitoring image of the work status of the work machine 80B.

[0103] [Fourth embodiment] Next, a fourth embodiment will be described. Note that the same components as those in the above-described embodiment are denoted by the same reference numerals, and redundant description will be omitted.

[0104] Figure 14 is a block diagram showing the functional configuration of a remote machine control device and a work machine according to a fourth embodiment. A remote machine control system 1C includes a remote machine control device 10C instead of the remote machine control device 10 shown in Figure 2. The remote machine control device 10C is obtained by adding, to the configuration of the remote machine control device 10 shown in Figure 2, a myoelectricity information acquisition unit 15 that acquires myoelectricity information from an operator, and a myoelectricity information processing unit 16 that performs predetermined signal processing on the acquired myoelectricity information.

[0105] The myoelectric potential information acquisition unit 15 is a plurality of myoelectric potential sensors attached to predetermined locations on the arm of an operator who operates the stick portion 11s of the operation unit 11. In this embodiment, the myoelectric potential information acquisition unit 15 is attached to six muscle bellies (two locations on the upper right upper arm and four locations on the forearm) of muscles used for movements such as elbow flexion and extension when operating the stick portion 11s. Specifically, the six muscle bellies are the biceps brachii, triceps brachii, palmaris longus, extensor carpi radialis longus, area near the supinator, and flexor carpi radialis.

[0106] 15 is a waveform diagram showing an example of myoelectric potential information acquired by the myoelectric potential information acquisition unit 15. The six pieces of myoelectric potential information acquired from the six muscle belly locations described above appear as a result of muscle belly activation during movements such as elbow flexion, elbow extension, wrist flexion, wrist tension, wrist extension, and wrist flexion. However, the myoelectric potential information itself is weak and is significantly affected by ambient electromagnetic noise and power supply hum noise, making it difficult to use the information as is.

[0107] Therefore, the myoelectric potential information processing unit 16 converts the six pieces of myoelectric potential information acquired by the myoelectric potential information acquisition unit 15 into respective muscle activity levels. The muscle activity levels are expressed based on a second-order delay model, which describes the mechanism from when muscle fibers are stimulated to when muscle tension is actually generated. Specifically, if the muscle activity level is a and the absolute value of the myoelectric potential information is u, the relationship between the two is expressed by equations (1) and (2).

[0108] The left side of equation (1) is the time differential value of the intermediate variable e, and τne in equation (1), τact and τdeac in equation (2) are predetermined time constants.

[0109] Furthermore, since the myoelectric potential information and muscle activity of each muscle change drastically over time and are significantly affected by external disturbances, there are many problems in using them directly to predict the control information. For this reason, in this embodiment, the myoelectric potential information processing unit 16 calculates the time pattern vector of muscle synergy.

[0110] A muscle synergy is a cooperative structure observed when multiple muscles are activated simultaneously, and is expressed using a temporal pattern (vector) consisting of multiple time-varying components and a spatial pattern (matrix) consisting of a set of vectors that summarize the contribution of each component to the activity intensity of each muscle. Specifically, when the temporal pattern vector of muscle synergy is H(t) and the spatial pattern matrix is ​​W, Equation (3) holds.

[0111] A(t)=W·H(t)+E(t) (3) where A(t) is the muscle activation vector and E(t) is the residual vector.

[0112] The number of components, i.e., the order k of H(t), is set so that the values ​​of the elements of E(t) are sufficiently small compared to the values ​​of the corresponding elements of A(t). In this embodiment, k=3 is adopted as the smallest order such that the values ​​of all elements of E(t) at any time t are less than 5% of the values ​​of the corresponding elements of A(t).

[0113] The six-dimensional muscle activity vector A(t) is obtained by converting six pieces of myoelectric potential information acquired from the arm when the stick 11s is operated using equations (1) and (2). Furthermore, this muscle activity vector A(t) can be expressed by equation (4) using a three-dimensional time pattern vector H(t) and a 6 × 3-dimensional spatial pattern matrix W.

[0114]

[0115] In equation (4), a1(t) to a6(t) on the left side are the elements of the muscle activity vector A(t) calculated from the EMG information at time t, and correspond to the muscle activity of the biceps brachii, triceps brachii, palmaris longus, extensor carpi radialis longus, supinator region, and flexor carpi radialis at time t, respectively. Also, h1(t) to h3(t) on the right side are the elements of the three-dimensional time pattern vector H(t) at time t, and hereinafter will be referred to as synergy 1, synergy 2, and synergy 3, respectively.

[0116] On the right-hand side, w11 to w63 and e1(t) to e6(t) are elements of the time-independent 6×3-dimensional spatial pattern matrix W and elements of the time-dependent residual vector E(t), respectively. The 6×3-dimensional spatial pattern matrix W is extracted using nonnegative matrix factorization (NMF) as a time-independent matrix having elements such that the elements of the residual vector E(t) are minimum at any time t.

[0117] 16 is a waveform diagram showing an example of each element of the extracted time pattern vector of muscle synergy. The future steering information prediction unit 25 shown in FIG. 14 predicts future steering information ξp(t+Δ), which is steering information after a first predetermined time has elapsed since the operation time t, by referring to not only the past operating forces F (plurality of sampling data) during a period from the operation time t to a second predetermined time before the operation time t but also the past muscle synergy time patterns (the same number of sampling data) during the same period.

[0118] (Learning Process of Future Prediction Model) In this embodiment, the machine learning device also performs machine learning processing of the future prediction model in the same manner as in steps S11 to S17 in Fig. 6. However, after step S11 in Fig. 6 is performed, the following processing is performed in steps S12 and S13.

[0119] In step S12, the machine learning device acquires, in addition to past operating forces (F(ta-w), ..., F(ta)) that have been sampled and measured and normalized within the range from the operation time ta to the time before w, three synergies (h1(ta-w), ..., h1(ta)), (h2(ta-w), ..., h2(ta)), (h3(ta-w), ..., h3(ta)) acquired within the same time range from Synergy 1, Synergy 2, and Synergy 3, which have been normalized to the range of -1.0 to 1.0 based on the maximum value of all absolute values.

[0120] In step S13, the machine learning device calculates (predicts) future operation information ξp(ta+Δ) using a future prediction model that inputs the operation information ξ(ta), past operation forces (F(ta-w), ..., F(ta)), and three past synergies. Thereafter, the processes from step S14 onward shown in FIG. 6 are performed.

[0121] FIG. 17 is a diagram comparing predictions of control information (operational force) with and without myoelectric potential information. Here, the prediction width Δ is set to 400 milliseconds. The diagram shows future control information (operational force) predicted using only the operation force for the control information (operational force) (Fx(t), Fy(t)) obtained by actual measurement, and future control information predicted using myoelectric potential information in addition to the operation force (both (Fxp(t) and Fyp(t))). Note that LSTM is again used as the machine learning model, and w is set to 2.0 seconds.

[0122] In FIG. 17, the upper row shows a comparison of the operating force Fx, with the left upper row showing the results when prediction is made using only the operating force, and the right upper row showing the results when prediction is made using myoelectric potential information in addition to the operating force. The lower row shows a comparison of the operating force Fy, with the left lower row showing the results when prediction is made using only the operating force, and the right lower row showing the results when prediction is made using myoelectric potential information in addition to the operating force.

[0123] 17, it can be seen that for both Fpx(t) and Fpy(t), the predicted future control information is closer to the actual measured value when myoelectric potential information is additionally used. In particular, in areas where there are large changes, there is a tendency for changes in control information to be predicted earlier when myoelectric potential information is additionally used. This is because, in the case of the right arm of a human body, changes in myoelectric potential are detected approximately 50 milliseconds before the occurrence of muscle tension.

[0124] Although the case where the prediction width Δ is 400 milliseconds is shown here, it has been found that when the prediction width is shorter, for example, up to 200 milliseconds, the actual measured value can be predicted almost correctly regardless of whether or not additional myoelectric potential information is used.

[0125] On the other hand, for predictions longer than that, for example, with a prediction width Δ of 600 milliseconds ahead, although the error increases, there is a tendency that the start position of the change in operating force can be predicted more accurately by additionally using myoelectric potential information. However, for prediction widths Δ of 800 milliseconds or more, the reproducibility deteriorates, and the results show that this method is not practical.

[0126] Other Embodiments The present invention is not limited to the above-described embodiments, but can also be applied to, for example, remote control of automobiles or construction machinery located in remote locations.

[0127] 1 and 2 has the operation unit 11 (FIG. 3) equipped with the stick unit 11s for operation in the above-described embodiment, but it may also have a ring-shaped handle, such as a steering wheel used in an automobile, instead of the stick unit 11s. In this case, the steering wheel has a neutral state, and when steered, a force is generated that tries to return it to the neutral state, so the present invention can be applied in the same way as in the above-described embodiment.

[0128] On the other hand, the work machine device 80 is applied as an automobile or construction machine itself located in a remote location. In this case, the working unit 83 corresponds to the steering mechanism of the automobile or the like, and the camera 84 captures images of the front direction of the automobile, etc. As a result, the monitor 14 of the remote machine control device 10 displays an image of the scenery that the driver can see through the windshield of the automobile or the like. The operator can then remotely control the automobile or the like located in a remote location with the same feeling as if they were actually driving the automobile or the like.

[0129] 1, 1B, 1C Remote machine control system 10 10C Remote machine control device 25, 25A Future operation information prediction unit 50 Network 80, 80B Work machine device

Claims

1. A remote machine control device for an operator to operate a control unit while viewing monitoring video to control a work machine at a remote location, comprising: an operating force acquisition unit that acquires the operating force applied to the operating unit; a future prediction model holding unit that holds a future prediction model, which is a machine learning model trained using, as teacher data, operation information at the time of operation of the operating unit, operation information at a time after a transmission delay time between the operation time and the work machine, and time series data of the operation force within a predetermined range in the past from the operation time; a prediction unit that predicts future operation information, which is future operation information after a time equivalent to the transmission delay time between the operation unit and the work machine, based on the operation information obtained from the operating unit operated by the operator, the operating force acquired by the operating force acquisition unit, and the future prediction model held in the future prediction model holding unit, using the operation time of the operating unit as a reference; and a transmission unit that transmits the future operation information predicted by the prediction unit to the work machine.

2. A remote machine control device as described in claim 1, further comprising an electromyogram information acquisition unit that acquires electromyogram information of the arm related to the operation of the operator, wherein the future prediction model is a machine learning model that is trained further using time series data of the electromyogram information, and the prediction unit predicts the future operation information further using the electromyogram information acquired by the electromyogram information acquisition unit.

3. A remote machine control device for an operator to operate a control unit while viewing monitoring video to control a work machine at a remote location, comprising: a transmission delay time measurement unit that measures a transmission delay time between the work machine and the operation unit based on the operation time on the control unit; a transmission delay time adjustment dial that continuously adjusts the transmission delay time measured by the transmission delay time measurement unit in accordance with the operation of the operator; a prediction unit that predicts future operation information, which is future operation information after the passage of a time equivalent to the transmission delay time adjusted by the transmission delay time adjustment dial, based on the operation time on the operation unit, based on the operation information obtained from the operation unit operated by the operator, specified information related to the operation of the operator, and a future prediction model; and a transmission unit that transmits the future operation information predicted by the prediction unit to the work machine.

4. A recording medium on which a program for causing a computer to function as each part of the remote machine control device described in claim 1 is recorded.

5. A program for causing a computer to function as each part of the remote machine control device according to claim 1.

6. A remote machine control system comprising: a remote machine control device according to claim 1 or claim 3; and a work machine device that receives the future control information transmitted from the remote machine control device and transmits the monitoring video to the remote machine control device.

Images