Information processing system

Abstract

Disclosed is an information processing system (S) including an operating-side device (1) and an operated-side device (2), the operating-side device (1) and the operated-side device (2) being capable of communicating with each other over a network, wherein the operating-side device (1) accepts input of information on the current position of the operating-side device (1) on the basis of an operation at the operating-side device (1) and transmits the position information to the operated-side device (2), the operated-side device (2) calculates a target position and an intensity of action by the operated-side device (2) on the basis of the position information and engages in action in accordance with the calculation results, the operated-side device (2) transmits information on the current position of the operated-side device (2) to the operating-side device (1) while engaged in action, and the operating-side device (1) adjusts the amount of position control using haptic information, including a target position and an intensity of action by the operating-side device (1), on the basis of the position information for the operated-side device (1).

Description

Information Processing Systems

[0001] The present disclosure relates to an information processing system.

[0002] Conventionally, remote control systems that remotely operate robotic arms and the like via a network are known (see, for example, Non-Patent Document 1). A typical remote control mechanism involves providing identical control objects for a leader (operating side) and a follower (operated side), and remote control is performed by synchronizing positions based on position information. When the leader is operated, the follower moves in accordance with the leader's movement based on the leader's position information. The follower's haptic information is fed back to the leader, and the leader operates the follower while checking the haptic sensations (resistance, etc.) experienced by the follower. In this case, position control is performed bidirectionally between the leader and follower based on position information, and operation and haptic feedback are achieved through position synchronization. This position control is performed at intervals of 1 ms or less.

[0003] Motionlib, Inc., "AbcCore, an IC chip that realizes Real Haptics (registered trademark)," [Online], 2022, [December 6, 2023], Internet <URL: https: / / www.motionlib.com / product / abc-core / >

[0004] When position synchronization is performed for both the leader and follower, delays occur when remote control is performed between distant locations. When transmitting information over a multiplexed network, unpredictable queuing delays occur in addition to propagation delays that depend on the transmission distance. If a large delay occurs due to congestion, old position information is exchanged through bidirectional position control, which may prevent operation from converging and cause oscillations. For example, if an operator releases their hand when a 100 ms delay is added to the relay network, both the leader and follower will exchange delayed and symmetrical position information. This causes both to try to synchronize to a slightly earlier state, resulting in oscillations.

[0005] In view of the above circumstances, an object of the present disclosure is to reduce the vibration phenomenon caused by network delays in remote control.

[0006] An information processing system according to one embodiment is an information processing system including an operating side device and an operated side device, wherein the operating side device and the operated side device are capable of communicating with each other via a network, the operating side device receives input of current position information of the operating side device from an operation on the operating side device and transmits the position information to the operated side device, the operated side device calculates a target position and strength of the operation of the operated side device from the position information and operates according to the calculation results, the operated side device transmits the current position information of the operated side device to the operating side device while operating, and the operating side device adjusts the amount of position control using force-tactile information including the target position and strength of the operation of the operating side device based on the position information of the operated side device.

[0007] According to the present disclosure, it is possible to reduce vibration phenomena caused by network delays in remote control.

[0008] FIG. 1 is a block diagram showing the configuration of an information processing system according to an embodiment. FIG. 2 is a block diagram showing the configuration of an operating side device. FIG. 3 is a diagram showing the configuration of an operated side device. FIG. 4 is a diagram showing a first example of a combination of an operating side device and an operated side device. FIG. 5 is a diagram showing a second example of a combination of an operating side device and an operated side device. FIG. 6 is a diagram showing a first diagram showing the positional relationship between an operating side device and an operated side device. FIG. 7 is a diagram showing a second diagram showing the positional relationship between an operating side device and an operated side device. FIG. 8 is a diagram showing a positional relationship between an operating side device and an operated side device. FIG. 9 is a diagram showing a positional relationship between an operating side device and an operated side device.

[0009] An embodiment will be described below with reference to the drawings. In each drawing, identical or corresponding parts are designated by the same reference numerals. In the description of this embodiment, the description of identical or corresponding parts will be omitted or simplified as appropriate.

[0010] 1 is a schematic diagram of an information processing system S according to this embodiment. The information processing system S includes an operating device 1 and an operated device 2 that can communicate with each other via a network NW. The operating device 1 is also referred to as a leader device. The operated device 2 is also referred to as a follower device. The network NW includes, for example, a mobile communication network, a fixed communication network, or the Internet.

[0011] The information processing system S is a remote control system in which an operating device 1 remotely operates an operated device 2 via a network NW. The operated device 2 is, for example, a robot. To perform precise operation, it is necessary to feed back the tactile sensation of the operated device 2 to the operating device 1, and for the operator of the operating device 1 to operate the operated device 2 while checking the tactile sensation of the operated device 2 in real time. The impact or resistance force when the operated device 2 grabs or hits an object is fed back to the operating device 1 in real time. The operator of the operating device 1 can operate the operated device 2 while receiving the feedback.

[0012] The network NW may include, for example, the Internet, at least one WAN, at least one MAN, or any combination thereof. "WAN" is an abbreviation for wide area network. "MAN" is an abbreviation for metropolitan area network. The network may include at least one wireless network, at least one optical network, or any combination thereof. The wireless network may be, for example, an ad-hoc network, a cellular network, a wireless LAN, a satellite communication network, or a terrestrial microwave network. "LAN" is an abbreviation for local area network.

[0013] 1 shows two operating side devices 1 and two operated side devices 2. However, the number of operating side devices 1 and two operated side devices 2 is not limited to this.

[0014] 2 illustrates the internal configuration of the operating-side device 1. The operating-side device 1 includes an operating-side interface 11, a position information detection unit 12, a motion calculation unit 13, a target position calculation unit 14, a relative position calculation unit 15, a reaction force information confirmation unit 16, a transmission unit 17, a reception unit 18, and a communication port 19. The components of the operating-side device 1 are connected to each other so that they can communicate with each other.

[0015] The operating-side interface 11 is a physical interface of the operating-side device 1 .

[0016] The position information detection unit 12 detects position information of the operating side device 1 (e.g., the operating side interface 11). The position information may include information indicating tilt or attitude, etc. The position information detection unit 12 may be, for example, a sensor. The action calculation unit 13 calculates the target position or strength of the action by the operating side device 1. The target position calculation unit 14 calculates the target position of the action by the operating side device 1. The relative position calculation unit 15 calculates the relative position (i.e., the position difference) between the operating side device 1 and the operated side device 2.

[0017] The transmitting section 17 transmits information relating to the operating side device 1 to the operated side device 2. The receiving section 18 receives information relating to the operated side device 2. The communication port 19 is an interface used for transmission and reception.

[0018] 3 illustrates the internal configuration of the operated side device 2. The operated side device 2 includes an operated side interface 21, a position information detection unit 22, a reaction force information detection unit 23, a motion calculation unit 24, a transmission unit 25, a reception unit 26, and a communication port 27. The components of the operated side device 2 are connected to each other so that they can communicate with each other.

[0019] The operated side interface 21 is a physical interface of the operated side device 2 .

[0020] The position information detection unit 22 detects position information of the operated side device 2. The position information detection unit 22 may be, for example, a sensor. The reaction force information detection unit 23 detects reaction force information of the operated side device 2. The reaction force information detection unit 23 may be, for example, a sensor. The movement calculation unit 24 calculates the target position and strength of the movement by the operated side device 2.

[0021] The transmitting section 25 transmits information relating to the operated-side device 2 to the operating-side device 1. The receiving section 26 receives information relating to the operating-side device 1. The communication port 27 is an interface used for transmitting and receiving information.

[0022] The operating side device 1 and the operated side device 2 may include a storage unit. The storage unit includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. A flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. A magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit functions as, for example, a main storage device, an auxiliary storage device, or a cache memory.

[0023] The functions of the motion calculation unit 13, the target position calculation unit 14, the relative position calculation unit 15, the reaction force information confirmation unit 16, the position information detection unit 22, the reaction force information detection unit 23, and the motion calculation unit 24 are executed by a control unit. The control unit includes, for example, one or more general-purpose processors including a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit may include one or more dedicated processors specialized for specific motions. Instead of including a processor, the control unit may include one or more dedicated circuits. The dedicated circuits may be, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit may include an ECU (Electronic Control Unit). The control unit controls the communication unit to send and receive any information.

[0024] 4 and 5 show examples of combinations of the operating-side device 1 and the operated-side device 2. The operating-side device 1 and the operated-side device 2 may have the same shape or similar shapes. The movement amount and torque due to control based on position information and control based on torque information may be the same between the operating-side device 1 and the operated-side device 2, or may be converted to different magnitudes. As shown in FIG. 4, the size of the operated-side device 2 is 10 times the size of the operating-side device 1, and the operated-side device 2 operates with a torque 10 times the torque applied to the operating-side device 1. The reaction force generated in the operated-side device 2 is fed back to the operating-side device 1 at one-tenth the magnitude.

[0025] The operating device 1 may have a different shape from the operated device 2. For example, as shown in FIG. 5 , the operating device 1 may have a shape similar to that of a game console controller, and the operated device 2 may have a shape similar to that of a robot. In this case, tactile information of different natures may be associated with each other, such as the tilt of a stick on a game console and the movement of a robot arm. Different actions may be associated with each other on the operating device 1 and the operated device 2, such as feeding back a rotational action in response to a linear action.

[0026] Each process executed by the information processing system S will be described in detail.

[0027] [First embodiment] Current position information of the operating side device 1 is input to the operating side interface 11 through operation on the operating side device 1. When the operating side device 1 accepts the input of the current position information, it transmits the position information to the operated side device 2 periodically or irregularly. The operated side device 2 calculates the target position and strength of the action of the operated side device 2 based on the position information of the operating side device 1, and operates according to the calculation result. While operating, the operated side device 2 transmits current position information of the operated side device 2 to the operating side device 1 periodically or irregularly. The operating side device 1 calculates haptic information including the target position and strength of the action based on the position information of the operated side device 2, and performs position control according to the calculation result. With this configuration, haptic information occurring in the operated side device 2 can be reflected in the operating side device 1 in real time, so the operator of the operating side device 1 can feel the impact or resistance force.

[0028] In the first embodiment, when the operating side device 1 receives feedback of position control from the operated side device 2, the operating side device 1 calculates the difference in position between the operating side device 1 and the operated side device 2 from the position information of the operated side device 2 in the relative position calculation unit 15. If the operating side device 1 determines that the difference is within a reference value, it does not execute position control. In other words, the position control is ignored and the amount of position control is adjusted to zero. The reference value is set as α here, but is not limited to this. If the operating side device 1 determines that the difference exceeds the reference value, it executes position control.

[0029] This configuration makes the operating side device 1 feel smoother and reduces slight vibrations caused by bidirectional position control. The larger α, the greater the tolerance to delays or operation speed, and the greater the ability to reduce vibrations. On the other hand, the larger α, the less tactile feedback there is, making it difficult to feel slight reaction forces. In this case, α needs to be adjusted depending on the situation.

[0030] For example, as shown in Figure 6, consider the case where the operating side device 1 is moved to position 61 by the operator. The movement distance here is 2α. The position of the operated side device 2 before the operating side device 1 moves is as shown in the figure. Control based on position information is performed in both directions. Since the difference in position between the operating side device 1 and the operated side device 2 is within α, the operating side device 1 is not subject to position control from the operated side device 2.

[0031] 7, the operated-side device 2 follows the operating-side device 1 and moves toward the current position 71 of the operating-side device 1. The operated-side device 2 is controlled with a delay of 1 ms or less. Since the difference between the positions of the operating-side device 1 and the operated-side device 2 is within α, the operating-side device 1 can operate without resistance.

[0032] As shown in Fig. 8, the operating side device 1 moves toward position 81. Position 81 is the same point as position 61. At this time, the operated side device 2 follows the operating side device 1 and moves toward position 82. Since the difference in position between the operating side device 1 and the operated side device 2 is within α, the operating side device 1 can move without resistance.

[0033] In the first embodiment, position control from the operated-side device 2 to the operating-side device 1 is not executed when the difference in position is within a reference value. Therefore, unless the operated-side device 2 hits an obstacle and stops, the operating-side device 1 does not receive feedback of the position control, and no vibration occurs.

[0034] [Second embodiment] In the first embodiment, the operating side device 1 ignores the feedback of position control from the operated side device 2 to the operating side device 1 if the position difference is within α. In this case, slight reaction force information is not fed back. However, in the second embodiment, if a reaction force is generated in the operated side device 2, position control is executed from the operated side device 2 even if the position difference is within α. Description of content common to the first embodiment will be omitted.

[0035] In the second embodiment, the operated-side device 2 detects the impact or resistance force when it comes into contact with an obstacle during an operation as a reaction force of the operation in the reaction force information detection unit 23. This reaction force is detected, for example, from a current value generated in a motor of the operated-side device 2 or a sensor. The operated-side device 2 transmits the reaction force information to the operating-side device 1.

[0036] When the operating side device 1 receives the reaction force information, the reaction force information confirmation unit 16 determines that a reaction force has occurred in the operated side device 2. When the operating side device 1 determines that a reaction force has occurred in the operated side device 2, it executes feedback of tactile information by position control, regardless of whether the difference in position between the operating side device 1 and the operated side device 2 is within α. When a reaction force is present, the force is not balanced due to an external force, so no vibration phenomenon occurs.

[0037] [Third Embodiment] In a third embodiment, when the target position calculation unit 14 of the operating side device 1 receives feedback of position control from the operated side device 2, it sets the midpoint between the positions of the operating side device 1 and the operated side device 2 as the target position for the operation of the operating side device 1. With this configuration, the position control is not performed symmetrically, and the control force from the operating side device 1 to the operated side device 2 is greater. Therefore, the vibration phenomenon converges and is reduced. In this way, the amount of position control is reduced.

[0038] For example, as shown in Fig. 9, consider a case where the operating device 1 is moved to a position 91 by the operator. As an example, the moving distance here is 2α. The position of the operated device 2 is as shown in the figure. Position control based on position information is performed in both directions.

[0039] 10, the operated-side device 2 moves toward the current position 101 of the operating-side device 1. In this case, position control is executed with a midpoint 102 between the positions of the operating-side device 1 and the operated-side device 2 as the target, and the operating-side device 1 moves toward the midpoint 102.

[0040] When the operating side device 1 is in the position shown in Fig. 11, the position of the operated side device 2 is as shown. In this case, position control is executed with midpoints 111 and 112 between the positions of the operating side device 1 and the operated side device 2 as targets. The operating side device 1 moves toward the midpoint 111. The operated side device 2 moves toward the midpoint 112. In this case, the magnitude of the position control from the operating side device 1 to the operated side device 2 is relatively larger than the magnitude of the position control from the operated side device 2 to the operating side device 1. As a result, the vibration converges.

[0041] Additionally or alternatively, the intermediate point does not have to be exactly in the middle between the operating device 1 and the operated device 2, but may be closer to either side.

[0042] 12 illustrates a process executed in the information processing system S. This process may be executed periodically.

[0043] In S1, the operating-side device 1 receives input of current position information of the operating-side device 1 from an operation on the operating-side device 1. In S2, the operating-side device 1 transmits the position information to the operated-side device 2.

[0044] In S3, the operated side device 2 calculates the target position and strength of the operation of the operated side device 2 from the position information. In S4, the operated side device 2 operates according to the calculation results. In S5, the operated side device 2 detects the current position information of the operated side device 2 while operating. In S6, the operated side device 2 transmits the position information to the operating side device 1.

[0045] In S7 and S8, the operating side device 1 adjusts the amount of position control using haptic information including the target position and strength of the operation of the operating side device 1, based on the position information of the operated side device 2. Specifically, in S7, the operating side device 1 determines from the position information of the operated side device 2 whether the difference between the position of the operating side device 1 and the position of the operated side device 2 is within a reference value. If the answer is No in S8, the operating side device 1 executes position control. If the answer is Yes in S7, the operating side device 1 does not execute S8.

[0046] As described above, according to this embodiment, the operating side device 1 accepts input of current position information of the operating side device 1 from operation on the operating side device 1 and transmits the position information to the operated side device 2. The operated side device 2 calculates the target position and strength of the action of the operated side device 2 from the position information and operates according to the calculation results. While operating, the operated side device 2 transmits the current position information of the operated side device 2 to the operating side device 1. Based on the position information of the operated side device 2, the operating side device 1 adjusts the amount of position control using haptic information including the target position and strength of the action of the operating side device 1. This configuration allows the operating side device 1 to reduce vibration phenomena caused by network delays.

[0047] According to this embodiment, the operating device 1 determines whether the difference between the positions of the operating device 1 and the operated device 2 is within a reference value based on the position information of the operated device 2. If it determines that the difference is within the reference value, it does not execute position control, but if it determines that the difference exceeds the reference value, it executes position control. With this configuration, the operating device 1 can provide a smoother feel when operating and reduce vibration.

[0048] Furthermore, according to this embodiment, the operating side device 1 determines whether the difference between the positions of the operating side device 1 and the operated side device 2 is within a reference value based on the position information of the operated side device 2, and when the operating side device 1 determines that a reaction force has occurred in the operated side device 2, it executes position control regardless of whether the difference is within the reference value. With this configuration, when a reaction force occurs, the operating side device 1 can reduce the vibration phenomenon while receiving feedback of tactile information. When a reaction force is present, the force is not balanced due to an external force, and the vibration phenomenon is reduced.

[0049] Furthermore, according to this embodiment, the operating-side device 1 sets the midpoint between the positions of the operating-side device 1 and the operated-side device 2 as the target position for the operation of the operating-side device 1, and executes position control. With this configuration, the position control is not symmetrical, and the control force from the operating-side device 1 to the operated-side device 2 is greater. As a result, the vibration phenomenon converges and is reduced.

[0050] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Two or more steps shown in the flowchart may be executed in parallel or in a different order, instead of being executed in chronological order as described, depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible within the scope of the present disclosure.

[0051] Each device constituting the information processing system S of this embodiment can also be realized by a computer and a program. Furthermore, in the above-described embodiment, a program that executes all or part of the functions or processes of each device can be recorded on a computer-readable recording medium or provided via a network. Computer-readable recording media include non-transitory computer-readable media, such as magnetic recording devices, optical disks, magneto-optical recording media, or semiconductor memories. The program can be distributed, for example, by selling, transferring, or lending portable recording media, such as Secure Digital (SD) cards, Digital Versatile Discs (DVDs), or Compact Disc Read Only Memory (CD-ROMs), on which the program is recorded. The program can also be distributed by storing the program in the storage of a server and transmitting it from the server to another computer via a network. The program can also be provided as a program product. The present disclosure can also be realized as a program executable by a processor. Some or all of the functions of each device can be implemented by programmable circuits or dedicated circuits. Some or all of the functions of each device can be implemented by hardware.

[0052] The following additional notes are further disclosed regarding the above-described embodiments.

[0053] (Supplementary Item 1) An information processing system including an operating side device and an operated side device, wherein the operating side device and the operated side device are capable of communicating with each other via a network, the operating side device accepts input of current position information of the operating side device from an operation on the operating side device and transmits the position information to the operated side device, the operated side device calculates a target position and strength of an action of the operated side device from the position information and operates in accordance with the calculation results, the operated side device transmits the current position information of the operated side device to the operating side device while operating, and the operating side device adjusts the amount of position control using force-tactile information including the target position and strength of the action of the operating side device based on the position information of the operated side device. (Supplementary Item 2) In the information processing system described in Supplementary Item 1, the operating side device determines from the position information of the operated side device whether a difference between the position of the operating side device and the position of the operated side device is within a reference value, does not execute the position control if it determines that the difference is within the reference value, and executes the position control if it determines that the difference exceeds the reference value. (Supplementary Item 3) In the information processing system described in Supplementary Item 1, the operating side device determines from the position information of the operated side device whether a difference between the position of the operating side device and the position of the operated side device is within a reference value, and executes the position control if it determines that a reaction force has been generated in the operated side device, regardless of whether the difference is within the reference value. (Supplementary Item 4) In the information processing system described in Supplementary Item 1, the operating side device sets an intermediate point between a position of the operating side device and a position of the operated side device as a target position for the operation of the operating side device, and executes the position control.

[0054] S Information Processing System

Claims

1. An information processing system including an operating-side device and an operated-side device, wherein the operating-side device and the operated-side device can communicate with each other via a network, the operating-side device receives an input of current position information of the operating-side device from an operation on the operating-side device and transmits the position information to the operated-side device, the operated-side device calculates a target position and intensity of the operation of the operated-side device from the position information and operates according to the calculation result, the operated-side device transmits current position information of the operated-side device to the operating-side device while operating, and the operating-side device adjusts an amount of position control using force feedback information including a target position and intensity of the operation of the operating-side device based on the position information of the operated-side device.

2. In the information processing system according to claim 1, the operating-side device determines whether a difference between the position of the operating-side device and the position of the operated-side device is within a reference value from the position information of the operated-side device. When it is determined that the difference is within the reference value, the position control is not executed. When it is determined that the difference exceeds the reference value, the position control is executed.

3. In the information processing system according to claim 1, the operating-side device determines whether a difference between the position of the operating-side device and the position of the operated-side device is within a reference value from the position information of the operated-side device. When the operating-side device determines that a reaction force has occurred in the operated-side device, the position control is executed regardless of whether the difference is within the reference value.

4. In the information processing system according to claim 1, the operating-side device sets a midpoint between the position of the operating-side device and the position of the operated-side device as the target position of the operation of the operating-side device and executes the position control.

Images