Remote machine control device and program and remote machine control system

Abstract

To provide a remote machine control device and program and remote machine control system capable of operating a machine without discomfort while monitoring a moving image of the machine installed at a remote place.SOLUTION: A remote machine control device 10 for allowing an operator to control a work machine device existing at a remote place while looking at a monitoring video includes a future control information prediction part 25 for predicting future control information being future control information after time passage corresponding to a transmission delay time with the work machine device 80 on the basis of an operation time in an operation part 11 on the basis of the operation information acquired from the operation part 11 operated by the operator, prescribed information related to the operation of the operator, and a future prediction model, and a transmission part 26 for transmitting the future control information predicted by the future control information prediction part 25 to the work machine device.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a remote machine control device and program, and a remote machine control system. [Background technology]

[0002] Conventionally, remote machine control systems have been known in which an operator remotely controls a machine installed in a remote location via a network. Specifically, the operator controls the machine by operating a control lever at hand while monitoring video of the machine's movement. In such remote machine control systems, there is a problem in that the operator feels a discrepancy between the monitoring video and the actual lever operation due to network transmission delays.

[0003] Therefore, a technology has been disclosed that reduces the impact of communication delays between an operation terminal (operator's operation lever) and an actual robot (machine) on the operator's operation (see Patent Document 1).

[0004] The technology in Patent Document 1 involves monitoring video footage of the movements of an actual robot operating at a remote location with a delay due to communication delays by modeling the actual robot and its surrounding objects, synthesizing predicted video that compensates for communication delays based on these models, and displaying this as the monitoring video. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-26028 Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology of Patent Document 1, the displayed monitoring image is highly dependent on the accuracy of the model that is the basis for the predictive synthesis. On the other hand, since the objects around the actual robot are not necessarily specific targets, it is difficult to always accurately model the objects around the robot, which results in the problem that an appropriate monitoring image cannot be displayed.

[0007] On the other hand, some of the inventors of the present application have proposed a technology that acquires myoelectric potential information from an electromyographic sensor attached to the operator's body and uses that myoelectric potential information to predict the operation information of a machine located in a remote location (Patent Application No. 2022-154191).

[0008] The above technique requires the operator to wear an electromyographic sensor on his or her body to acquire electromyographic information, which is a nuisance for the operator.

[0009] The present invention has been proposed in consideration of the above-mentioned situation, and provides a remote machine control device, program, and remote machine control system that enable a user to operate a machine installed in a remote location without feeling uncomfortable while monitoring the machine's movement video. [Means for solving the problem]

[0010] The remote machine control device of the present invention is a remote machine control device for an operator to control a work machine at a remote location while viewing monitoring footage, and is equipped with a future operation information prediction unit that predicts future operation information, which is future operation information after a time equivalent to the transmission delay time between the work machine and the operation unit has elapsed, based on operation information obtained from an operation unit operated by the operator, specified information related to the operator's operation, and a future prediction model, and a transmission unit that transmits the future operation information predicted by the future operation information prediction unit to the work machine.

[0011] The program according to the present invention causes a computer to function as each part of the remote machine control device.

[0012] The remote machine control system of the present invention comprises the remote machine control device and the work machine device that receives the future control information transmitted from the remote machine control device and transmits the monitoring video to the remote machine control device. [Effects of the Invention]

[0013] The present invention reduces the effects of transmission delays between machines installed in remote locations, allowing operators to operate the machines without feeling uncomfortable while viewing the monitoring video. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic diagram of a remote machine control system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the remote machine control device and the work machine device. [Figure 3] FIG. 2 is a schematic diagram illustrating an example of an operation unit. [Figure 4] 3 is a block diagram showing the functional configuration of a future driving information prediction unit; FIG. [Figure 5] 10 is a flowchart showing a prediction processing routine performed by a driving information prediction unit. [Figure 6] 10 is a flowchart showing the machine learning process of a future prediction model. [Figure 7] 4 is a waveform diagram showing an example of an operating force (operation information) obtained from an operating unit. FIG. [Figure 8] FIG. 2 is a schematic diagram showing a comparison between operation information (operation force) and future operation information. [Figure 9] FIG. 10 is a comparison diagram of actual operation information (operation force) and future operation information. [Figure 10] FIG. 10 is a block diagram showing a functional configuration of a future driving information prediction unit according to a second embodiment. [Figure 11] 10 is a flowchart showing a prediction processing routine performed by a driving information prediction unit according to the second embodiment. [Figure 12]10 is a flowchart showing a model update routine which is a subroutine of step S7. [Figure 13] FIG. 10 is a block diagram showing the functional configuration of a remote machine control device and a work machine device according to a third embodiment. [Figure 14] FIG. 10 is a block diagram showing the functional configuration of a remote machine control device and a work machine device according to a fourth embodiment. [Figure 15] 4 is a waveform diagram showing an example of myoelectric potential information acquired by a myoelectric potential information acquisition unit. FIG. [Figure 16] FIG. 10 is a waveform diagram showing an example of each element of an extracted muscle synergy time pattern vector. [Figure 17] 10 is a diagram comparing actual operation information and future operation information with and without myoelectric potential information. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0016] [First embodiment] (system configuration diagram) FIG. 1 is a schematic diagram of a remote machine control system 1 according to the first embodiment. The remote machine control system 1 includes a remote machine control device 10 that is operated by an operator, and a work machine device 80 that performs a predetermined task while communicating with the remote machine control device 10 via a network 50.

[0017] The work machine 80 is located at a location, for example, a remote location, away from the remote machine control device 10. The work machine 80 may be fixed to the location where it is located, or may be movable from the location where it is located.

[0018] The camera 84 of the work machine device 80 is installed in a position where it can capture an image of the working unit 83 performing a predetermined task. The camera 84 captures images of the work status of the working unit 83 and generates monitoring images. The work machine device 80 transmits the generated monitoring images to the remote machine control device 10 via the network 50.

[0019] The monitor 14 of the remote machine control device 10 displays the monitoring video transmitted from the work machine 80, i.e., the work status of the working unit 83. While viewing the monitoring video displayed on the monitor 14, the operator operates the operation unit 11 of the remote machine control device 10 to operate the work machine 80 (including the working unit 83 included therein) located in a remote location. In this embodiment, the operator's movement of the operation unit 11 is referred to as "operation," and the operator's movement of the work machine 80 (including the working unit 83) is referred to as "operation."

[0020] In this way, when the operator operates the operation unit 11, the remote machine control device 10 transmits control information to the work machine device 80. Then, when the work machine device 80 receives the control information, it performs an operation corresponding to the operation of the operation unit 11.

[0021] When information is transmitted between the remote machine control device 10 and the work machine 80, a transmission delay occurs due to the network 50. The transmission delay time associated with the information transmission from the remote machine control device 10 to the work machine 80 (outbound) is set to Δ1.

[0022] When the remote machine control device 10 transmits control information at time t1 and the work machine device 80 receives the control information at time t2, the following equation holds true. t2=t1+Δ1

[0023] Meanwhile, the transmission delay time associated with the information transmission from the work machine 80 to the remote machine control device 10 (return route) is assumed to be Δ2. When the work machine 80 transmits monitoring video of the work status of the working unit 83 captured by the camera 84 at time t'2, and the remote machine control device 10 receives and displays that monitoring video at time t3, the following equation holds: t3=t'2+Δ2

[0024] The return transmission delay time Δ2 includes not only the delay caused by the information transmission over the network 50, but also the delay caused by the image processing of the monitoring video performed before and after the information transmission. As a result, the transmission delay time Δ1 and the transmission delay time Δ2 do not necessarily coincide.

[0025] Here, assuming that t2=t'2, t3=t1+Δ1+Δ2. In other words, if the operator operates the operation unit 11 of the remote machine control device 10 at time t1, the monitor 14 displays the working status of the working unit 83 as a result of that operation at time t3.

[0026] FIG. 2 is a block diagram showing the functional configuration of the remote machine control device 10 and the work machine device 80. The remote machine control device 10 includes an operation unit 11 operated by the user, a control information prediction unit 12 that predicts future control information, a video receiving unit 13 that receives monitoring video, and a monitor 14 that displays the work status of the work machine device 80 as monitoring video.

[0027] 3 is a schematic diagram showing an example of the operation unit 11. The operation unit 11 is not particularly limited as long as it can be operated by an operator, but in this embodiment it is an operation lever. The operation unit 11 includes a stick unit 11s for operating the work machine device 80, and a dial adjustment unit 11d for manually adjusting a predicted time width Δ, which will be described later. The predicted time width Δ adjusted by the dial adjustment unit 11d is used by the operation information prediction unit 12.

[0028] The stick 11s is rod-shaped and stands upright perpendicular to the base when in a neutral state. The stick 11s can be tilted in any direction within the xy plane in response to an operator's operation. The angle at which the stick 11s is tilted (displaced) in any direction from the neutral state is called the displacement angle θ, and the projections of this angle onto the x-axis and y-axis, i.e., its x-component and y-component, are called the displacement angle θx and the displacement angle θy, respectively.

[0029] Spring-like members that can expand and contract in the x and y directions are attached to the base of the stick 11s. When the operator tilts the stick 11s from its neutral position by a displacement angle θ, the spring-like members apply forces to the stick 11s that are simply proportional to the displacement angles θx and θy, respectively, to return the stick 11s to its neutral position. The stick 11s is also provided with sensors that output signals corresponding to the expansion of the respective spring-like members.

[0030] When the stick unit 11s is tilted by a displacement angle θ by operation of the operator, the operation unit 11 supplies a signal corresponding to the displacement angle θ from the neutral state and signals corresponding to the extension of each spring-like member in the x and y directions to the operation information prediction unit 12. Hereinafter, any time when the operator continuously operates the operation unit 11 is defined as operation time t.

[0031] As shown in FIG. 2, the operation information prediction unit 12 includes a displacement information acquisition unit 21, a calculation unit 22, an operation force acquisition unit 23, a transmission delay time measurement unit 24, a future operation information prediction unit 25, and a transmission unit 26.

[0032] The displacement information acquisition unit 21 receives a signal corresponding to the displacement angle θ from the neutral state of the operation unit 11, and detects the displacement angle θ for each sampling period. The displacement information acquisition unit 21 supplies the detected displacement angle θ to the calculation unit 22.

[0033] The calculation unit 22 calculates the steering information ξ(θ) for each sampling period using the displacement angle θ supplied from the displacement information acquisition unit 21 as an input, and supplies the steering information ξ(θ) to the future steering information prediction unit 25. The steering information ξ(θ) is information for steering the work machine device 80, and is expressed as a predetermined function of the displacement angle θ.

[0034] The operating force acquisition unit 23 receives signals corresponding to the extensions in the x and y directions of the spring-like member provided in the operating unit 11, acquires the operating force F applied to the operating unit 11 by the operator for each sampling period, and supplies this to the future operation information prediction unit 25. Hereinafter, the operating force applied in the x direction is designated as Fx, and the operating force applied in the y direction is designated as Fy.

[0035] The transmission delay time measurement unit 24 measures a predicted time width Δ when predicting future control information. The predicted time width Δ is the sum of the transmission delay time Δ1 associated with the information transmission on the outbound route and the transmission delay time Δ2 associated with the information transmission on the return route.

[0036] The method for measuring the predicted time width Δ is not particularly limited as long as it can be measured by the transmission delay time measurement unit 24. The transmission delay time measurement unit 24 may provide a transmission delay time Δmeasure (fixed value) measured in advance in the preparation stage as the predicted time width Δ. Alternatively, the predicted time width Δ may be measured using an "echo request / echo reply" message of the Internet Control Message Protocol (ICMP), which is used in the Ping command that checks the connection status on the network.

[0037] Furthermore, as mentioned above, the transmission delay time Δ1 and the transmission delay time Δ2 do not necessarily coincide. Therefore, if the internal clocks of the remote machine control device 10 and the work machine 80 are synchronized with high precision (for example, on the order of sub-milliseconds) by sharing GPS information, for example, the following method is possible. That is, the transmission delay time measurement unit 24 may calculate the transmission delay times Δ1 and Δ2 using the transmission time and reception time (of information) recorded by the remote machine control device 10 and the work machine 80, respectively, and then measure the predicted time width Δ by taking the sum of these values.

[0038] However, if the predicted time width Δ measured as described above is used as is to predict the operation information, the monitoring image may show the work machine device 80 (for example, the working unit 83) reacting excessively to the operator's operation, which may cause the operator to feel uncomfortable. To reduce this sense of discomfort, when the operator operates the adjustment dial 11d of the operation unit 11 shown in Figure 3, the transmission delay time measurement unit 24 may adjust the size of the predicted time width Δ in accordance with the amount of operation.

[0039] The future control information prediction unit 25 predicts, for each sampling period, future control information ξp(t+Δ), which is control information after a first predetermined time has elapsed since the operation time t, based on the control information ξ(t) as a function of the displacement angle θ(t) at the operation time t. Note that the first predetermined time is equal to the predicted time width Δ measured by the transmission delay time measurement unit 24.

[0040] Specifically, the future operation information prediction unit 25 predicts future operation information ξp(t+Δ), which is the operation information after a first predetermined time has elapsed from the operation time t, by referring to past operation forces F (multiple sampling data) within the period from the operation time t to a second predetermined time before the operation time t, for the operation information ξ(t) at the operation time t.

[0041] The reason why future control information prediction unit 25 can predict future control information ξp(t+Δ) using sampling data of past operating force F is as follows. As described above, control information ξ(t) is a function of the displacement angle θ(t) of operation unit 11. The displacement angle θ(t) is determined by the balance between the force applied by the operator to operation unit 11 and the force caused by the spring-like member that the operator feels as a response from operation unit 11. For this reason, future control information prediction unit 25 can predict future control information ξp(t+Δ) with high accuracy by using the response of operation unit 11 up to that point, that is, sampling data of past operating force F. Specifically, future control information prediction unit 25 is configured as shown in FIG. 4.

[0042] FIG. 4 is a block diagram showing the functional configuration of the future driving information prediction unit 25. As shown in FIG. The future control information prediction unit 25 includes a future prediction model holding unit 31 that holds a future prediction model, a prediction unit 32 that predicts future control information ξp(t+Δ), and a control information holding unit 33 that holds control information ξ(t) at operation time t and future control information ξp(t+Δ).

[0043] The future prediction model held in the future prediction model holding unit 31 is, for example, a machine learning model that has been externally trained. The future prediction model has been trained by machine learning using a large amount of training data, and thereby establishes a relationship between the operation information ξ(t) at operation time t, the past operation force F (plurality of sampling data) within the period from the operation time t to a second predetermined time before, and the future operation information ξp(t+Δ).

[0044] The prediction unit 32 predicts future maneuver information ξp(t+Δ) using the future prediction model held in the future prediction model holding unit 31. Specifically, the prediction unit 32 inputs, into the future prediction model, the maneuver information ξ(t) at operation time t supplied from the calculation unit 22 shown in FIG. 2 and the past operation force F=(Fy, Fy) (plurality of sampling data within a period from the operation time t to a second predetermined time before) supplied from the operation force acquisition unit 23. Then, the prediction unit 32 outputs the information output from the future prediction model as future maneuver information ξp(t+Δ).

[0045] The operation information holding unit 33 holds the operation information ξ(t) at the operation time t supplied from the calculation unit 22 and the future operation information ξp(t) predicted by the prediction unit 32 in association with each other.

[0046] 2, the future steering information prediction unit 25 supplies the predicted future steering information ξp(t+Δ) to the transmission unit 26. The transmission unit 26 transmits the future steering information ξp(t+Δ) predicted by the future steering information prediction unit 25 to the work machine device 80 via the network 50.

[0047] In this way, the remote machine control device 10 transmits the future control information ξp(t+Δ) to the work machine device 80 at the operation time t. Here, the future control information ξp(t+Δ) refers to control information that is predicted to be generated by the operator in the future after the predicted time width Δ (=Δ1+Δ2) has elapsed from the operation time t.

[0048] The future manipulation information ξp(t+Δ) (=ξp(t+Δ1+Δ2)) transmitted from the remote machine manipulation device 10 is delayed by time Δ1 when passing through the network 50. Therefore, the future manipulation information received by the work machine device 80 at the operation time (t+Δ1) is ξp(t+Δ2).

[0049] As shown in Figure 2, the work machine device 80 includes a receiving unit 81 that receives future operation information ξp(t+Δ) from the remote machine operation device 10, a control unit 82 that generates machine control information from the received operation information, a working unit 83 that performs specified work in accordance with the machine control information generated by the control unit 82, a camera 84 that captures the work status of the working unit 83 and generates monitoring video, and a video transmitting unit 85 that transmits the monitoring video of the working unit 83 obtained by the camera 84.

[0050] The control unit 82 controls the working unit 83 using machine control information generated based on the future operation information ξp(t+Δ) received by the receiving unit 81. As a result, the working unit 83 performs a predetermined task based on the future operation information ξp(t+Δ2) after the time Δ1 has elapsed since the operation time t. The camera 84 captures images of the working status of the working unit 83. The video transmission unit 85 performs predetermined image processing, such as video compression processing, on the monitoring video of the working unit 83 captured by the camera 84, and transmits the obtained image-processed monitoring video to the remote machine control device 10.

[0051] The monitoring video transmitted from the work machine 80 is delayed by time Δ2 when it passes through the network 50 and reaches the remote machine control device 10. The video receiving unit 13 of the remote machine control device 10 then receives the monitoring video transmitted from the work machine 80, performs predetermined image processing on the video, and supplies the processed video to the monitor 14. The monitor 14 displays the monitoring video supplied from the video receiving unit 13.

[0052] The future operation information ξp(t+Δ1+Δ2) transmitted from the remote machine control device 10 arrives at the work machine 80 with a delay of time Δ1 as it passes through the network 50. As a result, the working unit 83 of the work machine 80 performs a predetermined task based on the future operation information ξp(t+Δ2) with respect to the operation time t of the operation unit 11, i.e., a task that is a time Δ2 in the future from the operation time t. Then, the monitoring video of the working unit 83 captured by the camera 84 is displayed on the monitor 14 of the remote machine control device 10 with a delay of time Δ2 as it passes through the network 50.

[0053] As a result, the monitor 14 displays a monitoring image of a predetermined task based on the future operation information ξp(t) at the operation time t of the operation unit 11, i.e., a task status that was predicted to occur at the same time as the operation time t. Therefore, the operator can operate the operation unit 11 without being affected by the transmission delay of the network 50, in other words, without feeling any discomfort from the monitoring image displayed on the monitor 14.

[0054] FIG. 5 is a flowchart showing a prediction processing routine performed by the driving information prediction unit 12. In step S1, the displacement information acquisition unit 21 detects whether or not the operation unit 11 has been operated. If the operation unit 11 has been operated, the process proceeds to the next step.

[0055] In step S2, the displacement information acquisition unit 21 acquires the displacement angle θ(t) of the operation unit 11 at the operation time t. The calculation unit 22 generates operation information ξ(t) based on a predetermined function that uses the displacement angle θ(t) acquired by the displacement information acquisition unit 21 as an input, and supplies the obtained operation information ξ(t) to the future operation information prediction unit 25.

[0056] The operating force acquisition unit 23 acquires the operating force applied to the operating unit 11 through a sensor attached to a spring-like member at the base of the stick portion 11s of the operating unit 11, and supplies the operating force to the future operation information prediction unit 25.

[0057] In step S3, the transmission delay time measurement unit 24 measures a predicted time width Δ corresponding to the transmission delay time caused by the round-trip information transmission between the remote machine control device 10 and the work machine device 80. The predicted time width Δ may be an actual measurement value measured by the above-mentioned conventional technology, or may be a preset fixed value.

[0058] In step S4, the future operation information prediction unit 25 predicts future operation information ξp(t+Δ), which is the operation information after a first predetermined time has elapsed from the operation time t, by referring to the past operation force F (multiple sampling data) within the period from the operation time t to the second predetermined time before the operation time t, for the operation information ξ(t) at the operation time t supplied from the calculation unit 22.

[0059] In step S5, the transmission unit transmits the future maneuvering information ξp(t+Δ) predicted by the future maneuvering information prediction unit 25 to the work machine device 80.

[0060] In step S6, it is determined whether or not the operation of the work machine device 80 has ended, and if it is determined that the operation has not ended, the process returns to step S1. If it is determined that the operation of the work machine device 80 has ended, the series of processes ends.

[0061] Next, we will explain the machine learning process of the future prediction model. The future prediction model is a machine learning model that is trained using the operation information ξ(t) and operation force F(t) at operation time t, which are obtained in advance by actual measurement, as training data.

[0062] (Learning process for future prediction model) 6 is a flowchart showing the machine learning process of a future prediction model. That is, a machine learning device (not shown) performs the machine learning process of a future prediction model according to the steps from step S11 onwards.

[0063] In step S11, the machine learning device sets an initial value w to the operation time ta and acquires the operation information ξ(ta), where w is the second predetermined time mentioned above.

[0064] In step S12, the machine learning device acquires past operation forces (F(ta-w), . . . , F(ta)) that are sampled and measured within a range from the operation time ta to a time before w.

[0065] In step S13, the machine learning device uses a future prediction model that inputs the operation information ξ(ta) and past operation forces (F(ta-w),...,F(ta)) to calculate (predict) future operation information ξp(ta+Δ), which is the operation information at the operation time (ta+Δ) that is a first predetermined time Δ after the operation time ta.

[0066] In step S14, the machine learning device acquires operation information ξ(ta+Δ) which is an actual measurement value at the operation time (ta+Δ).

[0067] In step S15, the machine learning device updates the internal parameters of the future prediction model in a direction that reduces the difference between the future maneuvering information ξp(ta+Δ) and the corresponding maneuvering information ξ(ta+Δ), which is the actual measured value as training data.

[0068] In step S16, the machine learning device increases the operation time ta by a time δ, where δ is the sampling time interval.

[0069] In step S17, the machine learning device ends the process if the actual measurement value ξ(ta) of the operation information at the operation time ta does not exist, and returns to step S12 if the actual measurement value ξ(ta) of the operation information at the operation time ta does exist.

[0070] As described above, in the remote machine control system 1 according to this embodiment, the remote machine control device 10 predicts and transmits future control information, which is control information for the future, by an amount corresponding to the round-trip transmission delay time, to the work machine 80 installed in a remote location. This allows the operator to operate the work machine 80 without feeling uncomfortable while checking the monitoring video of the work status of the work machine 80.

[0071] (Measurement example) Next, the calculation (prediction) of the future maneuver information ξp(t+Δ) using the actual measurement value of the maneuver information will be explained using a specific example. Although the control information ξ has been described as being expressed as a predetermined function of the displacement angle θ, in the following, to reduce the computational load, it is assumed to be simply proportional to the displacement angle θ, i.e., a linear function of the displacement angle θ. On the other hand, as described above, the operating force F is also simply proportional to the displacement angle θ. Therefore, when the constant k is used, the following relationship holds between the control information ξ and the operating force F: ξ=k·F

[0072] In this embodiment, k=1 is set to further simplify the calculation. That is, the operating force F is also used as the steering information ξ in its original form. Therefore, the steering information ξ(t) at time t is equivalent to the steering force F(t), and its x-component and y-component are also expressed as Fx(t) and Fy(t), respectively. The x-component and y-component of future steering information ξp(t+Δ), which is steering information after a predicted time width Δ from time t, are also expressed as Fpx(t+Δ) and Fpy(t+Δ), respectively. Here, it is assumed that the x-component and y-component of the steering information ξ (operating force F) can be handled independently of each other.

[0073] FIG. 7 is a waveform diagram showing an example of the operating force F (operation information ξ) obtained from the operating unit 11. The vertical axis represents the x-component (Fx) and y-component (Fy) of the operating force F output in sample units from the operating unit 11, normalized to a range of -1.0 to 1.0 based on the maximum absolute value of those components. The horizontal axis represents the time axis (seconds). Each thick solid line plots the first 1000 samples (20 seconds) sampled at a sampling frequency of 50 Hz.

[0074] The following mainly describes how to handle Fx, which is the x component of the maneuver information ξ, but Fy, which is the y component, can also be handled in the same way.

[0075] FIG. 8 is a diagram that schematically shows a comparison between the operation information ξ (operation force Fx) obtained by actual measurement and the future operation information ξp (future operation force Fpx) predicted using the same. The future operation information ξp(t+Δ) (future operation force Fpx(t+Δ)) is predicted from the operation information ξ(ta) (operation force Fx(ta)) at the operation time ta and sample data of the past operation force F (Fx(ta-w), . . . , Fx(ta)) within the period from the operation time ta to the time w before. In this embodiment, since the operation information ξ is equivalent to the operation force Fx, Fx(ta) is the operation information ξ(ta) actually measured at the operation time ta, and at the same time, it is also one of the sample data of the past operation force F used to predict ξp(t+Δ).

[0076] 9 is a diagram comparing the actual operation information ξ (operational force F) and the future operation information ξp (future operation force Fp) obtained through an experiment. Here, the prediction time width Δ, which is the first predetermined time described above, is set to 500 milliseconds, and the actually measured values ​​of the operation forces Fx(t) and Fy(t) are compared with the future operation forces Fpx(t) and Fpy(t) predicted using the future prediction model.

[0077] In the experiment shown in FIG. 9, a machine learning model that has undergone the machine learning process shown in FIG. 6, specifically, LSTM (Long Short Term Memory), was used as the future prediction model.

[0078] In addition, the acquisition range period w for the past operating force sampling data (Fx(tw),...,Fx(t)) and (Fy(tw),...,Fy(t)), which corresponds to the second predetermined period described above, was set to w = 2.0 seconds.

[0079] Furthermore, the sampling frequency of the actual control information ξ (operating force F) was set to 50 Hz. Therefore, the past operating force F required to predict the future control information ξ (future operating force F) was sampled at a time width of 20 milliseconds within a 2-second acquisition range, resulting in 100 samples of the past operating force F being input.

[0080] This indicates that, starting from a certain operation time t, the future operation information ξp 0.5 seconds after the operation time t is predicted using the operation force F sampled for the past 2.0 seconds. Therefore, as shown by the dotted line in Fig. 9, for example, the future operation force Fpx(t) first appears at the operation time t = 2.5 seconds.

[0081] In this embodiment, LSTM is used as an example of a machine learning model, but there is no particular limitation as long as the machine learning model can predict time-series data. For example, the machine learning model may be a Gated Recurrent Unit (GRU), which is a recurrent neural network, or a time convolution network.

[0082] In this embodiment, the x component and the y component of the steering information ξ (operating force F) are not dependent on each other and can be handled independently.

[0083] However, when an interaction between the x and y components of the control information ξ (operating force F) is expected, machine learning processing may be performed on the machine learning model for calculating Fpx(t+Δ) using not only the x component (Fx(tw), , Fx(t)) of the past operating force vector, but also its y component (Fy(tw), , Fy(t)) as training data.

[0084] 4 stores the operation information ξ(t) at operation time t and the future operation information ξp(t) predicted by the prediction unit 32 in association with the operation time t. Therefore, the information held in the operation information holding unit 33 may be retrieved to the outside of the system at a predetermined timing, and the retrieved information may be used as training data or the like to perform machine learning processing and update the internal parameters of the future prediction model. In this case, the future prediction model held in the future prediction model holding unit 31 may be replaced with the future prediction model updated as described above under predetermined conditions.

[0085] [Second embodiment] Next, a second embodiment will be described. The same components as those in the first embodiment are denoted by the same reference numerals, and redundant explanations will be omitted. In the second embodiment, a future operation information prediction unit 25A shown in the following FIG. 10 is used instead of the future operation information prediction unit 25 shown in FIG. 4.

[0086] 10 is a block diagram showing the functional configuration of a future steering information prediction unit 25A according to the second embodiment. The future steering information prediction unit 25A is obtained by adding a future prediction model update unit 34 to the configuration of the future steering information prediction unit 25 shown in FIG. The future prediction model update unit 34 compares the operation information ξ(t) at operation time t supplied from the calculation unit 22 with the future operation information ξp(t) predicted at operation time (t-Δ) and stored in the operation information storage unit 33, and updates the future prediction model stored in the future prediction model storage unit 31.

[0087] 11 is a flowchart showing a prediction processing routine by a driving control information predicting unit 12A (not shown) including a future driving control information predicting unit 25A. Here, the prediction processing routine executes steps S1 to S5 in the same manner as in FIG. 5, then executes a new step S7, and finally executes step S6.

[0088] In step S7, the future prediction model update unit 34 compares the operation information ξ(t) with the future operation information ξp(t) and updates the future prediction model held in the future prediction model holding unit 31. Specifically, the model update routine shown in Fig. 12 is executed.

[0089] FIG. 12 is a flowchart showing a model update routine, which is a subroutine of step S7 shown in FIG. In step S21, the future prediction model update unit 34 acquires the operation information ξ(t) at the operation time t supplied from the calculation unit 22.

[0090] In step S22, the future prediction model update unit 34 obtains future operation information ξp(t) at operation time t, predicted at operation time (t-Δ), which is a prediction time width Δ in the past from the operation time t, from the future operation information predicted in the past stored in the operation information storage unit 33.

[0091] In step S23, the future prediction model update unit 34 updates the internal parameters of the future prediction model stored in the future prediction model storage unit 31 so as to reduce the difference between the operation information ξ(t) and the future operation information ξp(t). Then, the future prediction model update unit 34 repeatedly executes steps S21 to S23 each time the operation time t advances.

[0092] However, if the future prediction model update unit 34 were to update the future prediction model every time sample data of the steering information ξ(t) and the future steering information ξp(t) is supplied, large-scale and high-speed computing power would be required, resulting in a large computational load. Therefore, the future prediction model update unit 34 may update the future prediction model for each sample data of the steering information ξ(t) and the future steering information ξp(t) every predetermined number of samples (e.g., 20 samples).

[0093] As described above, the remote machine control device 10 according to this embodiment, while predicting future operation information, updates the future prediction model at any time so that the previously predicted future operation information matches the actual operation information. Therefore, even if a change occurs in the work situation of the work machine device 80 and the prediction accuracy of the future operation information decreases, the discomfort felt by the operator who is operating the work machine device 80 while checking the monitoring video of the work situation can be gradually reduced.

[0094] [Third embodiment] Next, a third embodiment will be described. Note that the same components as those in the above-described embodiments are given the same reference numerals, and redundant explanations will be omitted.

[0095] Figure 13 is a block diagram showing the functional configuration of a remote machine control device and a work machine according to the third embodiment. The remote machine control system 1B is equipped with a work machine 80B instead of the work machine 80 shown in Figure 2. The work machine 80B is configured by adding a predicted image synthesis unit 86 that synthesizes predicted images to the configuration of the work machine 80 shown in Figure 2.

[0096] In the first embodiment, the remote machine control device 10 predicts future control information ξp(t+Δ), which is control information after a predicted time width Δ (=Δ1+Δ2) has elapsed from the operation time t, but in this embodiment, it predicts future control information ξp(t+Δ1), which is control information after only the outbound transmission delay time Δ1 has elapsed from the operation time t.

[0097] In the first embodiment, the transmission delay time measurement unit 24 sets the predicted time width Δ. In contrast to this, in the present embodiment, the transmission delay time measurement unit 24 sets the transmission delay time Δ1 instead of the predicted time width Δ. As a result, the remote machine control device 10 transmits, to the work machine device 80B, future control information ξp(t+Δ1), which is control information for the future by the outbound transmission delay time.

[0098] The future operation information ξp(t+Δ1) transmitted from the remote machine operating device 10 is delayed by time Δ1 when it passes through the network 50 and arrives at the work machine 80. As a result, the working unit 83 of the work machine 80B performs a predetermined task based on the future operation information ξp(t) at the operation time t of the operating unit 11, i.e., with almost no outbound transmission delay.

[0099] Camera 84 captures images of the work status of working unit 83. Predicted image synthesis unit 86 synthesizes predicted image of a future transmission delay time Δ2 with the monitoring image captured by camera 84, and supplies the synthesized predicted image as a monitoring image to image transmission unit 85. In other words, the monitoring image output from predicted image synthesis unit 86 is an image advanced into the future by transmission delay time Δ2.

[0100] The predicted video synthesis process by the predicted video synthesis unit 86 is a known technique and is disclosed, for example, in [B-7-20] "Low-delay, interactive, zero-latency video-somatic integrated network (4) Delay compensation by video prediction in remote work." The video transmission unit 85 performs predetermined image processing on the monitoring video synthesized with the predicted video, and transmits the obtained image-processed monitoring video to the remote machine control device 10.

[0101] The monitoring video transmitted from the work machine device 80B is delayed by a transmission delay time Δ2 when passing through the network 50, and arrives at the remote machine control device 10. The video receiving unit 13 of the remote machine control device 10 receives the monitoring video transmitted from the work machine device 80B. As a result, the monitor 14 displays the monitoring video with almost no transmission delay on the return path.

[0102] As described above, in the remote machine control system 1B according to this embodiment, the remote machine control device 10 predicts and transmits future control information ξp(t+Δ1), which is control information for the future by the outbound transmission delay time Δ1, to the work machine 80B installed in a remote location. Meanwhile, the work machine 80B synthesizes a predicted image for the future by the inbound transmission delay time Δ2 and transmits this as a monitoring image to the remote machine control device 10. This allows the operator to operate the work machine 80B without feeling uncomfortable while checking the monitoring image of the work status of the work machine 80B.

[0103] [Fourth embodiment] Next, a fourth embodiment will be described. Note that the same components as those in the above-described embodiments are given the same reference numerals, and redundant explanations will be omitted.

[0104] Figure 14 is a block diagram showing the functional configuration of a remote machine control device and a work machine according to a fourth embodiment. A remote machine control system 1C includes a remote machine control device 10C instead of the remote machine control device 10 shown in Figure 2. The remote machine control device 10C is obtained by adding, to the configuration of the remote machine control device 10 shown in Figure 2, a myoelectricity information acquisition unit 15 that acquires myoelectricity information from an operator, and a myoelectricity information processing unit 16 that performs predetermined signal processing on the acquired myoelectricity information.

[0105] The myoelectric potential information acquiring unit 15 is a plurality of myoelectric potential sensors attached to predetermined positions on the arm of an operator who operates the stick unit 11s of the operation unit 11. In this embodiment, the myoelectric potential information acquiring unit 15 is attached to six muscle bellies (two positions on the upper right upper arm and four positions on the forearm) of muscles used for movements such as elbow flexion and extension when operating the stick unit 11s. Specifically, the six muscle bellies are the biceps brachii, triceps brachii, palmaris longus, extensor carpi radialis longus, areas near the supinator, and flexor carpi radialis.

[0106] 15 is a waveform diagram showing an example of myoelectric potential information acquired by the myoelectric potential information acquisition unit 15. The six pieces of myoelectric potential information acquired from the six muscle belly locations described above appear as a result of muscle belly activation during movements such as elbow flexion, elbow extension, wrist flexion, wrist tension, wrist extension, and wrist flexion. However, the myoelectric potential information itself is weak and is significantly affected by surrounding electromagnetic noise and power supply hum noise, making it difficult to use the information as is.

[0107] Therefore, the myoelectric potential information processing unit 16 converts the six pieces of myoelectric potential information acquired by the myoelectric potential information acquisition unit 15 into respective muscle activity levels. The muscle activity levels are expressed based on a second-order delay model, which describes the mechanism from when muscle fibers are stimulated to when muscle tension is actually generated. Specifically, if the muscle activity level is a and the absolute value of the myoelectric potential information is u, the relationship between the two is expressed by equations (1) and (2).

[0108]

number

[0109] Furthermore, since the myoelectric potential information and muscle activity of each muscle change drastically over time and are significantly affected by external disturbances, there are many problems in using them directly to predict control information. For this reason, in this embodiment, the myoelectric potential information processor 16 calculates the time pattern vector of muscle synergy.

[0110] A muscle synergy is a cooperative structure observed when multiple muscles are activated simultaneously, and is expressed using a temporal pattern (vector) consisting of multiple time-varying components and a spatial pattern (matrix) consisting of a set of vectors that summarize the contribution of each component to the activity intensity of each muscle. Specifically, if the temporal pattern vector of muscle synergy is H(t) and the spatial pattern matrix is ​​W, then Equation (3) holds.

[0111] A(t)=W H(t)+E(t) (3) where A(t) is the muscle activation vector and E(t) is the residual vector.

[0112] The number of components, i.e., the order k of H(t), is set so that the values ​​of the elements of E(t) are sufficiently small compared to the values ​​of the corresponding elements of A(t). In this embodiment, k=3 is adopted as the smallest order such that the values ​​of all elements of E(t) at any time t are less than 5% of the values ​​of the corresponding elements of A(t).

[0113] The six-dimensional muscle activity vector A(t) is obtained by converting six pieces of myoelectric potential information acquired from the arm when operating the stick 11s using equations (1) and (2). Furthermore, this muscle activity vector A(t) can be expressed by equation (4) using a three-dimensional time pattern vector H(t) and a 6 × 3-dimensional spatial pattern matrix W.

[0114]

number

[0115] In equation (4), a1(t) to a6(t) on the left side are the elements of the muscle activity vector A(t) calculated from the EMG information at time t, and correspond to the muscle activity of the biceps brachii, triceps brachii, palmaris longus, extensor carpi radialis longus, supinator flexor, and flexor carpi radialis at time t, respectively. Also, h1(t) to h3(t) on the right side are the elements of the three-dimensional time pattern vector H(t) at time t, and hereinafter will be referred to as synergy 1, synergy 2, and synergy 3, respectively.

[0116] On the right-hand side, w11 to w63 and e1(t) to e6(t) are the elements of the time-independent 6 × 3-dimensional spatial pattern matrix W and the elements of the time-dependent residual vector E(t), respectively. Here, the 6 × 3-dimensional spatial pattern matrix W is extracted using nonnegative matrix factorization (NMF) as a time-independent matrix having elements that minimize the elements of the residual vector E(t) at any time t.

[0117] FIG. 16 is a waveform diagram showing an example of each element of the extracted muscle synergy time pattern vector. Then, the future steering information prediction unit 25 shown in FIG. 14 predicts future steering information ξp(t+Δ), which is steering information after a first predetermined time has elapsed since the operation time t, with reference to the steering information ξ(t) at the operation time t, in addition to the past operating forces F (plurality of sampling data) within a period from the operation time t to a second predetermined time before the operation time t, by referring to the time patterns of past muscle synergies (the same number of sampling data) within the same period.

[0118] (Learning process for future prediction model) In this embodiment, the machine learning device also performs machine learning processing of a future prediction model in the same manner as steps S11 to S17 in Fig. 6. However, after step S11 in Fig. 6 is performed, the following processing is performed in steps S12 and S13.

[0119] In step S12, the machine learning device acquires, in addition to the past operating forces (F(ta-w),...,F(ta)) that have been sampled and measured and normalized in the range from the operation time ta to the time before w, three synergies (h1(ta-w),...,h1(ta)), (h2(ta-w),...,h2(ta)), (h3(ta-w),...,h3(ta)) acquired in the same time range from Synergy 1, Synergy 2, and Synergy 3, which have been normalized to the range of -1.0 to 1.0 based on the maximum value of all absolute values.

[0120] In step S13, the machine learning device calculates (predicts) future maneuver information ξp(ta+Δ) using a future prediction model that uses as input the maneuver information ξ(ta), past maneuvering forces (F(ta-w), . . . , F(ta)), and three past synergies. Thereafter, the processes from step S14 onward shown in FIG. 6 are performed.

[0121] FIG. 17 compares predictions of control information (operating force) with and without myoelectric potential information. Here, the prediction width Δ is set to 400 milliseconds. The figure shows future control information (operating force) predicted using only the operating force for the control information (operating force) (Fx(t), Fy(t)) obtained through actual measurements, and future control information predicted using both the operating force and myoelectric potential information (both (Fxp(t) and Fyp(t))). Note that LSTM is again used as the machine learning model, with w set to 2.0 seconds.

[0122] In FIG. 17, the upper row is a comparison of the operating force Fx, with the left side of the upper row showing the results when prediction is made using only the operating force, and the right side showing the results when prediction is made using myoelectric potential information in addition to the operating force. The lower row is a comparison of the operating force Fy, with the left side showing the results when prediction is made using only the operating force, and the right side showing the results when prediction is made using myoelectric potential information in addition to the operating force.

[0123] 17 shows that the predicted future control information, for both Fpx(t) and Fpy(t), is closer to the actual measured value when myoelectric potential information is additionally used. In particular, in areas with large changes, the use of additional myoelectric potential information tends to enable earlier prediction of changes in control information. This is because, for the right arm of a human body, changes in myoelectric potential are detected approximately 50 milliseconds before the occurrence of muscle tension.

[0124] Although the case where the prediction width Δ=400 ms is shown here, it has been found that for shorter periods, for example, up to Δ=200 ms, the actual measured value can be predicted almost correctly, regardless of whether or not additional myoelectric potential information is used.

[0125] On the other hand, for predictions longer than that, for example, with a prediction width Δ of 600 milliseconds in the future, although the error increases, by additionally using EMG information, it tends to be possible to continue to more accurately predict the starting position of the change in operating force. However, for prediction widths of Δ of 800 milliseconds or more, the reproducibility deteriorates, and the results show that this method is not practical.

[0126] [Other embodiments] The present invention is not limited to the above-described embodiment, but can also be applied to, for example, remote control of automobiles or construction machinery located in remote locations.

[0127] 1 and 2 has an operation unit 11 (FIG. 3) equipped with a stick unit 11s for operation in the above-described embodiment, but it may also have a ring-shaped handle, such as a steering wheel used in an automobile, instead of the stick unit 11s. In this case, the steering wheel has a neutral state, and when steered, a force is generated that tries to return it to the neutral state, so the present invention can be applied in the same way as in the above-described embodiment.

[0128] On the other hand, the work machine device 80 is applied as an automobile or construction machine itself located in a remote location. In this case, the working unit 83 corresponds to the steering mechanism of the automobile or the like, and the camera 84 captures images of the front direction of the automobile, etc. As a result, the monitor 14 of the remote machine control device 10 displays an image of the scenery that the driver can see through the windshield of the automobile or the like. The operator can then remotely control the automobile or the like located in a remote location with the feeling that he or she is actually driving the automobile or the like. [Explanation of symbols]

[0129] 1,1B,1C Remote Machine Control System 10 10C Remote Machine Control Device 25, 25A Future Maneuver Information Prediction Unit 50 Network 80,80B Work machinery equipment

Claims

1. A remote machine control device for an operator to operate a work machine at a remote location while viewing a monitoring video, a future operation information prediction unit that predicts future operation information, which is future operation information after a time equivalent to a transmission delay time between the operation unit and the work machine device has elapsed, based on operation information obtained from an operation unit operated by the operator, predetermined information related to the operation of the operator, and a future prediction model, using the operation time of the operation unit as a reference; a transmission unit that transmits the future maneuvering information predicted by the future maneuvering information prediction unit to the work machine; A remote machine control device equipped with a

2. The future prediction model is a machine learning model trained using, as training data, the operation information at the operation time and the time after the transmission delay time from the operation time, and time series data of the predetermined information within a predetermined range in the past from the operation time.

2. The remote machine control device of claim 1.

3. an operation force acquisition unit that acquires an operation force applied to the operation unit; the future prediction model is the machine learning model trained using time-series data of the operating force as the time-series data of the predetermined information, The future operation information prediction unit predicts the future operation information using the operation force acquired by the operation force acquisition unit as the predetermined information.

3. The remote machine control device of claim 2.

4. a myoelectric potential information acquisition unit that acquires myoelectric potential information of an arm related to the operation of the operator, the future prediction model is the machine learning model trained by further using time series data of the myoelectric potential information as the time series data of the predetermined information, The future operation information prediction unit predicts the future operation information by further using the myoelectric potential information acquired by the myoelectric potential information acquisition unit as the predetermined information.

4. The remote machine control device of claim 3.

5. a future prediction model updating unit that updates the future prediction model based on the operation information at the operation time obtained from the operation unit and the future operation information corresponding to the operation time predicted in the past by the future operation information prediction unit; 3. The remote machine control device of claim 2, further comprising:

6. Further comprising a transmission delay time measurement unit for measuring the transmission delay time, The future steering information prediction unit predicts the future steering information after a time equivalent to the transmission delay time measured by the transmission delay time measurement unit has elapsed.

2. The remote machine control device of claim 1.

7. The transmission delay time measurement unit measures, as the transmission delay time, a round-trip transmission delay time from transmitting the future operation information to the work machine until the monitoring image is returned from the work machine.

7. The remote machine control device of claim 6.

8. a transmission delay time adjusting unit that adjusts the transmission delay time in response to an operation by the operator; 7. The remote machine control device of claim 6, further comprising:

9. A program for causing a computer to function as each part of the remote machine control device according to claim 1.

10. A remote machine control device according to claim 1; the work machine device that receives the future operation information transmitted from the remote machine control device and transmits the monitoring video to the remote machine control device; A remote machine control system equipped with

Images