Terminal, management server, communication system, method, and program

Abstract

The present disclosure provides a terminal, a management server, a communication system, a method, and a program with which it is possible to improve communication performance of haptic data. A terminal according to an embodiment of the present disclosure comprises: a reception unit that receives haptic data from another terminal connected via a network; a generation unit that, on the basis of an index indicating the communication state of the haptic data, generates instruction information for causing a management server on the network to execute control related to communication of the haptic data with the other terminal; and a transmission unit that transmits the instruction information to the management server.

Description

Terminal, management server, communication system, method and program

[0001] The present disclosure relates to a terminal, a management server, a communication system, a method, and a program.

[0002] Systems capable of providing various services to users have been developed in the field of communication technology. For example, Patent Literature 1 discloses a connection management system having a concierge assistant service for handling a wide range of topics and user intents via a common interface. The connection management system interprets the intent of natural language communication received from a user device and automatically determines a social network member endpoint that will receive the natural language communication based on the intent.

[0003] Special Publication No. 2022-502977

[0004] In recent years, there has been progress in technology for transmitting haptic data, which is data related to at least one of tactile and kinesthetic sensations, from a transmitting device to a receiving device located at a distance. The receiving device then transmits the haptic data to the user via an interface, allowing the user to experience virtual reality related to tactile sensations.

[0005] For example, in a system that performs remote operation, when a user performs remote operation on one device, data related to the remote operation instruction is transmitted from one device to another device. The other device performs an operation based on the instruction. The other device then transmits haptic data generated by the operation to the first device. The user can experience the haptic data through the interface, giving the user the sensation of actually performing some action on the spot. This allows the user to perform the desired operation.

[0006] However, when the distance between the transmitting device and the receiving device is long and the timing of data reception is delayed, or when the communication environment is degraded, it may be difficult for the user to feel as if they are actually performing the action in real time, and accurate remote control may not be possible. The technology described in Patent Document 1 does not address this issue because it does not target haptic technology.

[0007] An example of an object of the present disclosure is to provide a terminal, a management server, a communication system, a method, and a program that can improve communication performance of haptic data. Note that this object is only one of multiple objects that multiple embodiments disclosed herein aim to achieve. Other objects or problems and novel features will become apparent from the description of this specification or the accompanying drawings.

[0008] A terminal according to one aspect of the present disclosure includes a receiving unit that receives haptic data from other terminals connected via a network, a generating unit that generates instruction information addressed to a management server on the network based on an indicator indicating the communication status of the haptic data to execute control regarding communication of the haptic data with the other terminals, and a transmitting unit that transmits the instruction information to the management server.

[0009] A management server according to one aspect of the present disclosure is connected to a first terminal via a network and includes a receiving unit that receives instruction information from a second terminal that receives haptic data from the first terminal to execute control regarding communication of the haptic data with the first terminal, and a control execution unit that executes control regarding the communication of the haptic data based on the instruction information.

[0010] A communication system according to one aspect of the present disclosure includes a terminal and a management server. The terminal includes a receiving unit that receives haptic data from another terminal connected via a network, a generating unit that generates instruction information for the management server on the network based on an indicator of a communication status of the haptic data to execute control regarding communication of the haptic data with the other terminal, and a transmitting unit that transmits the instruction information to the management server. The management server includes a receiving unit that receives the instruction information from the terminal and a control executing unit that executes control regarding communication of the haptic data based on the instruction information.

[0011] A method according to one aspect of the present disclosure involves a computer receiving haptic data from another terminal connected via a network, generating instruction information for a management server on the network to execute control regarding communication of the haptic data with the other terminal based on an indicator indicating the communication status of the haptic data, and transmitting the instruction information to the management server.

[0012] A method according to another aspect of the present disclosure involves a computer receiving instruction information from a second terminal connected to a first terminal via a network and receiving haptic data from the first terminal, for executing control over communication of the haptic data with the first terminal, and executing control over the communication of the haptic data based on the instruction information.

[0013] A program according to one aspect of the present disclosure causes a computer to receive haptic data from another terminal connected via a network, generate instruction information addressed to a management server on the network based on an indicator indicating the communication status of the haptic data, for controlling communication of the haptic data with the other terminal, and send the instruction information to the management server.

[0014] A program according to another aspect of the present disclosure causes a computer to receive instruction information from a second terminal connected to a first terminal via a network and receiving haptic data from the first terminal, for executing control over communication of the haptic data with the first terminal, and execute control over the communication of the haptic data based on the instruction information.

[0015] According to the present disclosure, it is possible to provide a terminal, a management server, a communication system, a method, and a program that can improve the communication performance of haptic data.

[0016] 1 is a block diagram illustrating an example of a terminal according to the present disclosure. FIG. 2 is a flowchart illustrating an example of a representative process of a terminal according to the present disclosure. FIG. 3 is a block diagram illustrating an example of a management server according to the present disclosure. FIG. 4 is a flowchart illustrating an example of a representative process of a management server according to the present disclosure. FIG. 5 is a block diagram illustrating an example of a remote control system according to the present disclosure. FIG. 6 is a diagram illustrating an example of a haptic encoder selection scheme according to the present disclosure. FIG. 7 is a diagram illustrating a different example of a haptic encoder selection scheme according to the present disclosure. FIG. 8 is a diagram illustrating a specific example of a SEEHC header. FIG. 9 is a table illustrating SEEHC header coding. FIG. 10 is a diagram illustrating an example of a payload configuration used in a SEEHC header. FIG. 11 is a diagram illustrating an example of a payload configuration of a feedback bitstream corresponding to a forward bitstream. FIG. 11 is a diagram illustrating another specific example of a SEEHC header. FIG. 12 is a diagram illustrating another example of a payload configuration used in a SEEHC header. FIG. 13 is a diagram illustrating another example of a payload configuration of a feedback bitstream corresponding to a forward bitstream. FIG. 14 is a table illustrating actions associated with the SEEHC scheme when there is no delay. FIG. 15 is a table illustrating actions associated with the SEEHC scheme when there is a delay. FIG. 16 is a sequence diagram illustrating an SE server discovery procedure initiated and executed by a leader terminal. FIG. 17 is a sequence diagram illustrating an SE server discovery procedure initiated and executed by a follower terminal. It is an image diagram showing the training of the SE server 140 using the expert profile.It is a block diagram showing the effect of the present disclosure.It is a block diagram showing an example of the hardware configuration of the information processing device.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following descriptions and drawings in the embodiments have been omitted or simplified as appropriate for clarity of explanation. For example, in each drawing, identical or corresponding elements are given the same reference numerals, and duplicate explanations are omitted as necessary for clarity of explanation. It goes without saying that the drawings referred to for explanation in each embodiment can also be applied to other embodiments. Furthermore, in this disclosure, unless otherwise specified, when "at least one of" is defined for multiple items, the definition may mean any one item or any multiple items (including all items).

[0018] First Embodiment (1A) Hereinafter, a first embodiment of the present disclosure will be described. In this (1A), a terminal will be described.

[0019] 1 is a block diagram showing an example of a terminal. The terminal 10 is any type of computer including a receiving unit 11, a generating unit 12, and a transmitting unit 13. Each unit (means) of the terminal 10 is controlled by a control unit (controller) not shown. Each unit will be described below.

[0020] The receiving unit 11 receives haptic data from another terminal connected via a network. As described above, haptic data is data related to at least one of the sense of touch and the sense of movement. The haptic data transmitted by the other terminal is generated, for example, by an action performed by a human or a machine (e.g., a robot) on the other terminal. The haptic data acquired by the other terminal is transmitted to the terminal 10 through communication such as remote control.

[0021] The network connecting the terminal 10 and other terminals may be any type of network. A management server that manages communication related to haptic data between the terminal 10 and other terminals is provided on the network.

[0022] The generation unit 12 generates instruction information for a management server on the network to execute control regarding communication of haptic data with other terminals based on an indicator indicating the communication status of the haptic data received by the receiving unit 11.

[0023] The index indicating the communication state of the haptic data is any qualitative or quantitative index related to the communication state. Examples of the index include, but are not limited to, QoE (Quality of Experience), QoS (Quality of Service), packet loss, and transmission delay. The index may be acquired by the terminal 10 analyzing the haptic data received by the receiving unit 11. Alternatively, when a user of the terminal 10 experiences haptic data, the user may input the QoE related to the data into the terminal 10, which allows the terminal 10 to acquire the QoE. A specific example of this will be described later in the second embodiment.

[0024] Examples of instruction information generated by the generation unit 12 include the following: (A) instruction information for predicting network parameters between the terminal 10 and another terminal and controlling communication between the terminal 10 and the other terminal based on the predicted parameters; (B) instruction information for predicting haptic data to be generated at the other terminal and controlling the transmission of the haptic data to the terminal by modifying the content of the haptic data based on the predicted haptic data. The network parameters predicted by the management server in (A) may be, but are not limited to, at least one of packet loss, packet error, CRC (Cyclic Redundancy Check) error, error rate, bit rate, jitter, transmission delay, transmission speed, connection loss, bandwidth, capacity, load balancing, HAV data configuration, data stream, data flow, active session, retransmission, and codec selection. In (B), the management server may predict, for example, what type of haptic data will be generated as a result of a human or machine action being performed at the other terminal. The management server can insert data generated as a result of prediction into haptic data acquired from another terminal and transmitted to the terminal 10, and transmit the data to the terminal 10. However, the target predicted in (B) is not limited to this.

[0025] The transmitting unit 13 transmits the instruction information to the management server. The transmitting unit 13 is configured as an interface used for wireless or wired communication with other terminals.

[0026] 2 is a flowchart showing an example of a typical process of the terminal 10. The outline of the process of the terminal 10 will be explained with reference to this flowchart. Note that the details of each process are as described above, and therefore will not be explained again.

[0027] First, the receiving unit 11 receives haptic data from another terminal connected via a network (step S11). The generating unit 12 generates instruction information for changing control over communication of haptic data with another terminal based on an index indicating the communication status of the haptic data (step S12). Then, the transmitting unit 13 transmits the instruction information to the management server (step S13).

[0028] As described above, the terminal 10 can generate instruction information based on an index indicating the communication status of haptic data and transmit the instruction information to the management server. Therefore, the terminal 10 can improve the communication performance of haptic data through the control of the management server.

[0029] Furthermore, by generating and transmitting instruction information, the terminal 10 can have the management server predict network parameters and control communication based on the predicted parameters. Therefore, through the control of the management server, the terminal 10 can improve the communication performance of haptic data so that the network parameters are favorable.

[0030] Furthermore, by generating and transmitting instruction information, terminal 10 can have the management server predict haptic data to be generated on another terminal, and based on the predicted haptic data, change the content of the haptic data to be transmitted to the terminal. This allows terminal 10, through the control of the management server, to ensure that the haptic data it receives is closer to the content generated on the other terminal and originally intended to be transmitted to terminal 10. This allows terminal 10 to improve haptic data communication performance so that terminal 10 can receive haptic data that is closer to the original data.

[0031] (1B) Next, in (1B), the management server will be described. Fig. 3 is a block diagram showing an example of the management server. The management server 20 is any type of computer equipped with a receiving unit 21 and a control execution unit 22. Each unit (each means) of the management server 20 is controlled by a control unit (controller) not shown. Each unit will be described below.

[0032] The receiving unit 21 is connected to a first terminal via a network and receives, from a second terminal that receives haptic data from the first terminal, instruction information for changing control over communication of haptic data with the first terminal. Specific descriptions of the haptic data, instruction information, and network are the same as those in (1A), and therefore will not be repeated here.

[0033] The control execution unit 22 executes control related to communication of haptic data based on the instruction information received by the receiving unit 21. When the receiving unit 21 receives the instruction information shown in (A), the control execution unit 22 predicts parameters related to the network between the first terminal and the second terminal and controls communication between the first terminal and the second terminal based on the predicted parameters. Furthermore, when the receiving unit 21 receives the instruction information shown in (B), the control execution unit 22 predicts haptic data to be generated on the first terminal side and, based on the predicted haptic data, controls the haptic data to be transmitted to the second terminal so as to change the content of the haptic data. However, the control executed by the control execution unit 22 is not limited to this. Furthermore, when the control execution unit 22 executes control, the management server 20 may transmit information indicating the content of the executed control to at least one of the first terminal or the second terminal.

[0034] 4 is a flowchart showing an example of a typical process of the management server 20. An overview of the process of the management server 20 will be explained with reference to this flowchart. Note that the details of each process are as described above, and therefore will not be explained here.

[0035] First, the receiving unit 21 receives instruction information from the second terminal for executing control regarding communication of haptic data with the first terminal (step S21). The control executing unit 22 executes control regarding communication of haptic data based on the instruction information (step S22).

[0036] As described above, the management server 20 executes control related to the communication of haptic data based on instruction information received from the terminal. Therefore, the management server 20 can improve the communication performance of haptic data in response to instructions from the terminal.

[0037] Embodiment 2 In the following embodiment 2, specific examples of the terminal and management server described in embodiment 1 are disclosed. However, the specific examples of the terminal and management server shown in embodiment 1 are not limited to those shown below. Furthermore, the configurations and processes described below are examples and are not limited to these. It goes without saying that all or any part of the configuration described in embodiment 2 can be applied to embodiment 1 as appropriate, and the drawings shown in embodiment 2 can be applied to embodiment 1 as appropriate.

[0038] [Configuration Description] Fig. 5 is a block diagram showing an example of a remote control system. The remote control system 100 is a wide area communication network system including a leader terminal 110, follower terminals 120, a network 130, and an SE (Support Engine) server 140. The leader terminal 110 and follower terminals 120 correspond to the terminals 10 in the first embodiment, and the SE server 140 corresponds to the management server 20 in the first embodiment.

[0039] In the example of FIG. 5 , the teleoperation system 100 is a system that supports medical procedures, such as surgery, in a remote location. For example, the follower terminal 120 is a terminal located at the location where the surgery is actually performed on the patient, and the leader terminal 110 is a terminal located at the location where the surgeon (i.e., the user of the leader terminal 110) performs the surgery. As the surgeon performs the surgical movements, the leader terminal 110 transmits instructions related to the movements to the follower terminal 120. The follower terminal 120 controls a surgical robot connected to the follower terminal 120 based on the received instructions. In this way, the surgical robot operates in accordance with the surgeon's movements and performs the surgery on the patient. The surgery targets any part of the body (including, for example, the brain, nerves, etc.). Furthermore, the teleoperation system 100 is not limited to surgery, but can also be applied to any medical procedure that invasively affects the patient's body, such as biopsies, injections, and cancer treatments. However, the application of the teleoperation system 100 is not limited to this. The leader terminal 110 may also transmit instructions related to any action performed by the user to the follower terminal 120, not limited to user movements.

[0040] The leader terminal 110 includes a physical environment 111, a virtual environment 112, a HAV (Haptic Audio Video) data processing unit 113, an encoder / decoder 114, and a QoE acquisition unit 115. Each element of the leader terminal 110 will be described below.

[0041] The leader terminal 110 is configured to perform its processing using the physical environment 111 or the virtual environment 112. Note that the leader terminal 110 may be configured to have either the physical environment 111 or the virtual environment 112, rather than both.

[0042] The HAV data processing unit 113 generates HAV data based on data acquired from a sensor connected to the reader terminal 110. The generated HAV data includes haptic data, audio data, and video data. The sensor is attached to, for example, the surgeon's hand, acquires data on the surgeon's hand movement, and transmits it to the HAV data processing unit 113. The HAV data processing unit 113 processes the received data to generate HAV data. However, data may be transmitted to the HAV data processing unit 113 from any interface, not limited to a sensor. Furthermore, the HAV data may include haptic data and audio data or video data, or may be data including haptic data without including audio data or video data.

[0043] Furthermore, the HAV data processing unit 113 generates a SEEHC header that includes information for selecting a haptic encoder to be used for communicating HAV data and for issuing instructions to the SE server 140, as will be described later.

[0044] The encoder / decoder 114 is a unit that has both the functions of an encoder and a decoder. The encoder / decoder 114 encodes the HAV data generated by the HAV data processing unit 113 for transmission. The leader terminal 110 transmits the encoded HAV data to the follower terminals 120 via the network 130.

[0045] The encoder / decoder 114 also decodes the HAV data transmitted from the follower terminal 120 and received via a receiving unit (not shown) of the leader terminal 110. The encoder / decoder 114 outputs the decoded HAV data to an actuator attached to an instrument such as a surgical instrument or joystick held by the surgeon, or to a glove or the like worn by the surgeon. When the actuator operates in response to the HAV data, tactile sensations such as tactile sensations and force sensations are transmitted to the surgeon's hand. In this way, haptic data such as tactile sensations and force sensations generated in the robot arm on the follower terminal 120 as a result of the surgeon's surgery is fed back to the surgeon. In other words, the teleoperation system 100 can be said to be a system capable of performing bilateral control.

[0046] The encoder / decoder 114 supports, for example, No Delay Haptic Codec (NDHC), With Delay Haptic Codec (WDHP), and SEEHC formats. Details of how each format is selected will be described later.

[0047] The QoE acquisition unit 115 acquires QoE information on the leader terminal 110 side in HAV data communication (hereinafter also referred to as HAV communication) between the leader terminal 110 and the follower terminals 120. A user near the leader terminal 110, such as a surgeon, may manually input QoE information into the leader terminal 110, thereby allowing the QoE acquisition unit 115 to acquire the QoE information. The user inputs their own evaluation of the HAV data as the QoE to the QoE acquisition unit 115. Note that the QoE perceived by the user may vary depending on the presence or absence of communication delay or the degree of delay, or fluctuations in parameters related to communication quality other than delay. Alternatively, the QoE acquisition unit 115 may automatically acquire the QoE information. In this case, the QoE acquisition unit 115 can acquire the QoE information using any method, such as QoE-related measurements, a QoE point system, or MOS (Mean Opinion Score). In this way, the QoE acquisition unit 115 can monitor the QoE information. Furthermore, the QoE acquisition unit 115 can also report the status of HAV communication to the SE server 140 by transmitting QoE information to the SE server 140. The processing of the SE server 140 will be described later.

[0048] The follower terminal 120 includes a physical environment 121, a virtual environment 122, an HAV data processing unit 123, an encoder / decoder 124, and a QoE acquisition unit 125. Each element of the follower terminal 120 will be described below. Note that the same points as those in the description of the elements of the leader terminal 110 will be omitted as appropriate.

[0049] The follower terminal 120 is configured to perform its processing using a physical environment 121 or a virtual environment 122. Note that the follower terminal 120 may be configured to have either the physical environment 121 or the virtual environment 122, rather than both. The physical environment 121 and the virtual environment 122 are environments equivalent to the physical environment 111 and the virtual environment 112 of the leader terminal 110, respectively.

[0050] The HAV data processing unit 123 generates HAV data based on data acquired from a sensor connected to the follower terminal 120. The sensor is attached to, for example, a robot arm that performs surgery, and acquires data such as the tactile sensation and force sensation of the part that the robot arm comes into contact with, and transmits the data to the HAV data processing unit 123. The HAV data processing unit 123 processes the received data to generate HAV data.

[0051] Furthermore, the HAV data processing unit 123 generates a SEEHC header that includes information for selecting a haptic encoder to be used for communicating HAV data and instruction information for the SE server 140, as will be described later.

[0052] The encoder / decoder 124 is a unit that has both the functions of an encoder and a decoder. The encoder / decoder 124 encodes the HAV data generated by the HAV data processing unit 123 for transmission. The follower terminal 120 transmits the encoded HAV data to the leader terminal 110 via the network 130. The process performed by the leader terminal 110 on the transmitted HAV data has already been described.

[0053] The encoder / decoder 124 also decodes the HAV data transmitted from the leader terminal 110 and received via a receiving unit (not shown) of the follower terminal 120. The encoder / decoder 124 decodes the HAV data transmitted from the leader terminal 110. The encoder / decoder 124 outputs the decoded HAV data to an actuator attached to a robot arm that performs surgery. The actuator operates in accordance with the HAV data, causing the robot arm to perform surgery in accordance with the operation of a surgeon located in a remote location.

[0054] Like the encoder / decoder 114, the encoder / decoder 124 supports, for example, the NDHC, WDHP, and SEEHC formats.

[0055] The QoE acquisition unit 125 acquires QoE information on the follower terminal 120 side in HAV communication between the leader terminal 110 and the follower terminal 120. A person in the vicinity of the follower terminal 120 may manually input the QoE information into the follower terminal 120, thereby allowing the QoE acquisition unit 125 to acquire the QoE information. Alternatively, the QoE acquisition unit 125 may automatically acquire the QoE information. The method by which the QoE acquisition unit 125 automatically acquires the QoE information is similar to that of the QoE acquisition unit 115, and therefore will not be described here. In this way, the QoE acquisition unit 125 can monitor the QoE information. Furthermore, the QoE acquisition unit 125 can also report the status of the HAV communication to the SE server 140 by transmitting the QoE information to the SE server 140.

[0056] The network 130 interconnects the leader terminal 110, the follower terminal 120, and the SE server 140. The network 130 may be a wide area network such as 5G (5th Generation) or 6G (6th Generation), but the type of the network 130 is not limited thereto.

[0057] The network 130 includes a first network 131 and a second network 132. The first network 131 and the second network 132 include network interfaces, modules, and the like necessary for communication. The first network 131 is a network that does not cause delays that could interfere with remote surgery in HAV communication between the leader terminal 110 and the follower terminal 120. The first network 131 is, for example, a short-distance network or has a wireless interface optimized for HAV communication. On the other hand, the second network 132 is a network that may cause delays that could interfere with remote surgery in HAV communication between the leader terminal 110 and the follower terminal 120. The second network 132 is, for example, a long-distance network or has a wireless interface that is inefficient for HAV communication. Either the first network 131 or the second network 132 is used for communication between the leader terminal 110 and the follower terminal 120.

[0058] In addition, the network 130 can transfer HAV data sent from either the leader terminal 110 or the follower terminal 120 to the SE server 140 depending on the settings of the leader terminal 110 or the follower terminal 120 described below.

[0059] The SE server 140 is a server that controls HAV communication between the leader terminal 110 and the follower terminal 120, and includes a first functional unit 141, a second functional unit 142, a first model 143, a second model 144, and an interface 145. The SE server 140 is provided to reduce latency of a network used for HAV communication and improve the QoE of the remote operation system 100. Note that although only one SE server 140 is illustrated in FIG. 5 , multiple SE servers 140 may be provided in the remote operation system 100. Furthermore, the first functional unit 141 and the second functional unit 142 correspond to the control execution unit 22 according to the first embodiment, and the interface 145 corresponds to the receiving unit 21 according to the first embodiment.

[0060] The first functional unit 141 predicts network parameters specific to HAV communication for optimization. The network parameters include, for example, at least one of packet loss, packet error, CRC error, error rate, bit rate, jitter, transmission speed, transmission delay, connection loss, bandwidth, capacity, load balancing, HAV data configuration, data stream, data flow, active sessions, retransmission, codec selection, and encoding / decoding parameters. In this case, the first functional unit 141 can control the HAV communication based on the predicted network parameters. For example, the first functional unit 141 may set parameters for remote rendering in the HAV communication or set outsourcing of computation related to the HAV data. The first functional unit 141 can also predict multiple different parameters and control the HAV communication based on the predicted parameters. Furthermore, when multiple SE servers 140 are provided, different parameters may be predicted and controlled by the first functional unit 141 of each SE server 140.

[0061] The second functional unit 142 predicts at least one of the behavior of the user of the leader terminal 110 and haptic data detected by a sensor of the robot of the follower terminal 120. This behavior prediction is performed to improve the QoE of at least one of the leader terminal 110 and the follower terminal 120 in HAV communication. For example, the second functional unit 142 can predict the next (future) behavior of the surgeon on the leader terminal 110 side, thereby controlling the robot arm on the follower terminal 120 side to reduce the response time until the surgeon's behavior is reflected in the robot arm. The second functional unit 142 can also predict the haptic data that the robot sensor will detect next (in the future). This allows the second functional unit 142 to control the robot arm to reduce the response time until the surgeon can feel the tactile sensations and force sensations from the patient's body that occur in response to the surgeon's surgery. This improves the QoE in communication. The second functional unit 142 can perform prediction processing on the order of milliseconds.

[0062] The first model 143 is a pre-trained AI (Artificial Intelligence) model such as a neural network. The first function unit 141 inputs HAV data acquired by the SE server 140 to the first model 143. In response to this input data, the first model 143 outputs network parameters specific to HAV communication.

[0063] Like the first model 143, the second model 144 is also a pre-trained AI model such as a neural network. The second functional unit 142 inputs HAV data acquired by the SE server 140 to the second model 144. In response to this input data, the second model 144 outputs at least one of a prediction result of human behavior based on the HAV data and haptic data detected by a sensor of the robot.

[0064] Any AI-based prediction model can be applied to the first model 143 and the second model 144. Applicable AI-based prediction models include efficient models such as a convolutional neural network (CNN) and a recurrent neural network (RNN). Detailed training methods for the first model 143 and the second model 144 will be described later.

[0065] Furthermore, the first functional unit 141 and the second functional unit 142 of the SE server 140 can be individually set to enable or disable their functions as needed. Furthermore, the first functional unit 141 and the second functional unit 142 may execute at least one of generating and updating the first model 143 and the second model 144, respectively.

[0066] The interface 145 is an interface that connects the SE server 140 to the leader terminal 110 and the follower terminals 120 via the network 130. The interface 145 is, for example, an API (application programming interface), a service API, or the like. The SE server 140 receives HAV data transmitted from at least one of the leader terminal 110 and the follower terminal 120 via the interface 145. The interface 145 can also receive a SEEHC header from at least one of the leader terminal 110 and the follower terminal 120. The interface 145 can also transmit HAV data to at least one of the leader terminal 110 and the follower terminal 120.

[0067] Next, we will further explain the details of the processing of each part of the SE server 140. The SE server 140 can execute at least one of the following two modes based on the SEEHC sent from the leader terminal 110 or the follower terminal 120 to the SE server 140.

[0068] (1) SE server 140 may transmit to follower terminal 120 the HAV data that was sent from leader terminal 110 to follower terminal 120 and then transferred to SE server 140 without changing the content of the data. Conversely, SE server 140 may transmit to leader terminal 110 the HAV data that was sent from follower terminal 120 to leader terminal 110 and then transferred to SE server 140 without changing the content of the data. In other words, SE server 140 enters a passive mode in which it does not change the content of the HAV data based on the SEEHC.

[0069] Here, the SEEHC received by the SE server 140 includes information instructing the first function unit 141 on what kind of prediction to make. Based on the instruction information, the first function unit 141 may predict optimal network parameters for HAV communication and control the network parameters according to the predicted values. Through this process, the first function unit 141 can improve data transmission or traffic performance and improve the QoE experienced by the user. Therefore, remote control can be stably realized. This can be achieved regardless of whether the HAV data is transmitted from the leader terminal 110 to the follower terminal 120 or from the follower terminal 120 to the leader terminal 110.

[0070] Note that a plurality of SE servers 140 may be provided in the remote control system 100. In this case, the parameters that the first functional units 141 in each SE server 140 can predict may be different. If the parameters that the first functional units 141 can predict are different, the leader terminal 110 or the follower terminal 120 may include information indicating the function of the first functional unit 141 to be used (i.e., the parameters to be predicted) in the SEEHC. This SEEHC setting is realized as part of the encoding function of the encoder / decoder 114 or 124. Of the SE servers 140 that receive the SEEHC, only those SE servers 140 that can realize the functions specified in the SEEHC with their first functional units 141 execute the prediction process in their first functional units 141. By performing the above process, even if a plurality of SE servers 140 are provided, only the most appropriate SE server 140 is selected in terms of the processing to be executed. Details of the SEEHC will be described later.

[0071] (2) SE server 140 may change the content of HAV data that was sent from leader terminal 110 to follower terminal 120 and then transferred to SE server 140, and then transmit the changed data to follower terminal 120. Conversely, SE server 140 may change the content of HAV data that was sent from follower terminal 120 to leader terminal 110 and then transferred to SE server 140, and then transmit the changed data to leader terminal 110. In other words, SE server 140 enters an active mode in which it changes the content of the HAV data based on the SEEHC.

[0072] As described above, the second functional unit 142 can predict user behavior using the second model 144. Based on the prediction result, the second functional unit 142 includes data related to the predicted user behavior in the HAV data transmitted to the follower terminal 120. The follower terminal 120 performs remote operation reflecting the HAV data containing the predicted user behavior. This can improve the QoE in communications. For example, the second functional unit 142 adds data related to the predicted user behavior to the HAV data transferred from the leader terminal 110 so as to reduce the response time until the surgeon's behavior is reflected in the robot arm on the follower terminal 120 side. The second functional unit 142 then transmits the HAV data to the follower terminal 120.

[0073] The second functional unit 142 can also predict haptic data detected by the robot sensor of the follower device 120. Based on this prediction result, the second functional unit 142 includes the predicted haptic data in the HAV data transmitted to the leader device 110. The user of the leader device 110 experiences a virtual reality in which the HAV data containing the predicted haptic data is reflected. As a result, even if the data transmitted from the follower device 120 is insufficient for remote operation, the second functional unit 142 can add the missing data to the HAV data. This reduces the user's perceived disruption to remote operation and improves the QoE in communication. For example, the second functional unit 142 adds the predicted haptic data to the HAV data transmitted from the follower device 120 so as to reduce the response time between the haptic data detected by the robot sensor and the sensation felt by the surgeon. The second functional unit 142 then transmits the HAV data to the leader device 110.

[0074] Even if a network delay occurs, the second function unit 142 can make a prediction about the human or robot and provide immediate feedback so that the prediction result is reflected in the remote control data before new HAV data arrives at the SE server 140. This makes it possible to suppress the effects of network delays on remote control and maintain a high QoE.

[0075] However, in parallel with the above processing of the second functional unit 142, the first functional unit 141 may predict optimal network parameters for HAV communication and control the network parameters according to the predicted values, as described in (1). Also, in (1), the first functional unit 141 may not predict optimal network parameters for HAV communication based on instruction information such as SEEHC, and may not control the network parameters according to the predicted values. Also, processing that can be performed in parallel with the processing of the second functional unit 142 is not limited to the processing of the first functional unit 141.

[0076] The HAV data is data communicated between the leader terminal 110 and the follower terminal 120 by remote control, and the above-described processing can be realized by setting this data to reach the SE server 140. Furthermore, the SE server 140 can execute either processing (1) or (2) on at least one of the HAV data transmitted from the leader terminal 110 and the HAV data transmitted from the follower terminal 120.

[0077] In particular, the second functional unit 142 can improve the QoE of remote control by identifying communication restrictions in the TI (Tactile Internet).

[0078] Next, a sequence for selecting an appropriate codec scheme for HAV data, executed by the leader terminal 110 or the follower terminal 120, is defined. The codec scheme selects an appropriate coding procedure for haptic data and the presence of network delay. As described below, the codec scheme is applicable to both cases with and without network delay. Although the following example will be described mainly with the leader terminal 110, the follower terminal 120 can also perform similar processing.

[0079] 6 shows an example of a haptic encoder selection scheme. First, the HAV data processing unit 113 determines whether the HAV data received by the reader terminal 110 is kinesthetic data (step S31). Kinesthetic data is data including data related to at least one of force, torque, and pressure. If the HAV data is not kinesthetic data, the HAV data is tactile data. Tactile data is data related to a human's perception of the surface of an object through their senses (e.g., data indicating the feel of the hand).

[0080] If the HAV data received by the leader terminal 110 is kinesthetic data (Yes in step S31), the HAV data processing unit 113 determines whether the HAV data was transmitted via the first network 131 based on the selection of the encoder / decoder 114 used (step S32). As described above, the first network 131 is a network that does not cause delays in HAV communication that could interfere with telesurgery. On the other hand, the second network 132, which is not the first network 131, is a network that may cause delays in HAV communication that could interfere with telesurgery.

[0081] When the HAV data is transmitted via the first network 131, i.e., when the transmission is deemed to be without delay (Yes in step S32), the HAV data processor 113 selects NDHC as the encoder (step S33). NDHC is based on a Just Noticeable Difference (JND) encoder to reduce redundant data traffic. Furthermore, the HAV data processor 113 can train and optimize the NDHC encoder using network dynamics (step S34) to make encoding of the HAV data more efficient. Note that the network dynamics indicates changes over time in any parameter related to the communication quality of the HAV data detected by the HAV data processor 113 during the communication of the HAV data between the leader terminal 110 and the follower terminal 120. The communication quality parameters include, but are not limited to, jitter, packet loss, packet errors, transmission delay, and bandwidth. After training, the HAV data processing unit 113 can further train and optimize the NDHC encoder by utilizing feedback information from the network channel that is fed back to the NDHC encoder.

[0082] If the HAV data is transmitted via the second network 132, i.e., if the transmission is deemed to have a delay (No in step S32), the HAV data processing unit 113 selects WDHP as the encoder (step S35). WDHP is based on a combination of the Time-Domain Passivity Approach (TDPA) and a perceptual deadband (DB) coding scheme. TDPA encoding ensures passivity in a two-port network, including communication channels that may cause communication interference. Specifically, the signal transmissions of the leader and follower are attenuated to dissipate excess output energy, thereby supporting system stability and ensuring passivity.

[0083] If network dynamics are important, the HAV data processor 113 uses the WDHP scheme to train and optimize the encoding scheme in the presence of network delays and other signal transmission impairments using information about the network dynamics (step S36). After training, the HAV data processor 113 can further train and optimize the WDHP encoder by using feedback information from the network channel that is fed back to the WDHP encoder.

[0084] If the HAV data received by the leader terminal 110 is not kinesthetic data, i.e., if the HAV data is tactile data (No in step S31), the HAV data processor 113 selects TC (Tactile Codec) as the encoder (step S37). Haptic data has fewer communication requirements than kinesthetic data. Therefore, the HAV data processor 113 sets the sampling rate of TC by selecting a simple sampling scheme (step S38).

[0085] The HAV data processor 113 also trains and optimizes the encoding scheme based on network dynamics (step S39). After training, the HAV data processor 113 can further train and optimize the TC encoder by utilizing feedback information from the network channel that is fed back to the TC encoder.

[0086] In this way, the HAV data processing unit 113 determines whether the HAV data is kinesthetic data or haptic data, and functions as a change unit that changes the encoder used to communicate the HAV data based on the determination result. The HAV data processing unit 113 can select an appropriate encoder depending on the type of HAV data. This allows the HAV data processing unit 113 to accurately reproduce the data received by the leader terminal 110 for the user, leading to improved QoE.

[0087] Furthermore, if the HAV data is kinesthetic data, the HAV data processing unit 113 can further determine the degree of delay occurring in the network used for communication and change the encoder used for communicating the HAV data based on the determined degree of delay. Therefore, the HAV data processing unit 113 can improve the accuracy of reproducing the data received by the leader terminal 110 for the user, leading to further improvement in QoE.

[0088] The HAV data processing unit 113 may perform the following process to further optimize the encoding of haptic data when delays occur in the network.

[0089] FIG. 7 shows a different example of a haptic encoder selection scheme. In FIG. 7, if a transmission delay is deemed to exist, the selected codec is changed depending on whether the delay is large or small. Note that the explanation of steps S31 to S34 and S37 to S39 in FIG. 7 is omitted because they are the same as those in FIG. 6. In the following example, the leader terminal 110 will be mainly described, but similar processing can also be performed in the follower terminal 120.

[0090] When the HAV data is transmitted via the second network 132, i.e., when it is determined that there is a delay in the transmission (No in step S32), the HAV data processing unit 113 determines whether the transmission delay is large (step S40). For example, the HAV data processing unit 113 may determine whether the transmission delay is 1 ms or less, and determine that the delay is large if the delay is greater than 1 ms, and that the delay is medium if the delay is 1 ms or less. Note that a transmission delay of 1 ms or less means that the distance between the leader terminal 110 and the follower terminal 120 is approximately 150 km or less (approximately 300 km or less round trip distance).

[0091] If the transmission delay is not large (No in step S40), the HAV data processing unit 113 selects WDHC as the encoder (step S41). Furthermore, the HAV data processing unit 113 can train and optimize the WDHC encoder using network dynamics (step S42), thereby making the encoding of HAV data more efficient. After training, the HAV data processing unit 113 can further train and optimize the WDHC encoder by utilizing feedback information from the network channel that is fed back to the WDHC encoder. This allows the HAV data processing unit 113 to further improve the stability of remote control over a wide area network.

[0092] On the other hand, if the transmission delay is large (Yes in step S40), the HAV data processing unit 113 selects SEEHC as the encoder (step S43). SEEHC is a more preferable encoder than WDHC for solving delay issues when the transmission delay is large. As a result of this process, the SE server 140 operates the second model 144 as a prediction model that supports SEEHC. This SEEHC enables the necessary functional units in the SE server 140.

[0093] Furthermore, the HAV data processing unit 113 can train and optimize the SEEHC encoder using network dynamics (step S44) to make encoding of HAV data more efficient. After training, the HAV data processing unit 113 can further train and optimize the SEEHC encoder by utilizing feedback information from the network channel that is fed back to the SEEHC encoder. This allows the HAV data processing unit 113 to improve the stability of remote operation over a wide area network.

[0094] In steps S32 and S40, the HAV data processing unit 113 can use any existing technology to determine whether a delay is occurring in the network or whether the delay is significant. For example, when HAV data is being transmitted between the leader terminal 110 and the follower terminal 120, the leader terminal 110 internally stores TFNs (Time Frame Numbers) related to the communication. The HAV data processing unit 113 measures the TFNs at regular time intervals, for example, and detects the amount of change in the TFNs to determine whether a delay has occurred and the extent of the delay. However, the communication parameters used by the HAV data processing unit 113 for the determination are not limited to the TFNs.

[0095] Note that steps S33, S38, and S41 relate to network coding. Also, step S43 is not the only step in which the necessary functional units in the SE server 140 are enabled. For example, as a result of the processing in step S38, the SE server 140 can predict at least one of network parameters and user behavior using at least one of the first functional unit 141 and the second functional unit 142.

[0096] 6 and 7 can also be executed in the follower terminal 120. As a result of this processing, the SE server 140 can use at least one of the first functional unit 141 and the second functional unit 142 to predict at least one of the network parameters and the haptic data generated on the robot side.

[0097] The HAV data processing unit 113 may determine the encoder selection scheme shown in Figure 6 or Figure 7 using a trained model trained by using data including information indicating the type of HAV data and the degree of delay, and the corresponding encoder type (correct label) as training data.

[0098] Next, the SEEHC header transmitted from a transmitting unit (not shown) of the leader terminal 110 or the follower terminal 120 to the SE server 140 will be described. For example, as a result of selecting SEEHC by the processing of step S43 in FIG. 7 , the HAV data processing unit 113 or the HAV data processing unit 123 generates and transmits a SEEHC header according to the selection result. This enables the SE server 140 to operate the second model 144 and make predictions regarding the user or the robot. Note that even if an encoder other than SEEHC is selected in FIG. 7 , the leader terminal 110 or the follower terminal 120 can transmit a SEEHC header according to the selection result to the SE server 140. When transmitting HAV data, the HAV data processing unit 113 or the HAV data processing unit 123 can attach any of the following types of SEEHC headers to the HAV data and transmit the HAV data with the SEEHC header attached. In the following example, the leader terminal 110 will be mainly described, but the follower terminal 120 can also execute the same processing.

[0099] 8 shows a specific example of a SEEHC header (codec header) used when the HAV data is kinesthetic data and it is determined in step S32 of FIG. 7 that there is no delay. The SEEHC header includes a T field H1, an F field H2, an X field H3, and an S field H4. The bit size of each field corresponds to delay information in the system. The delay information indicates, for example, whether there is a delay in the system.

[0100] FIG. 9 is a table illustrating the SEEHC header coding corresponding to FIG. 8. The example of the SEEHC header shown in FIG. 9 is used when the HAV data received by the leader terminal 110 in FIG. 7 is kinesthetic data (Yes in step S31). In other words, when the HAV data is kinesthetic data, the SEEHC header shown in FIG. 9 can be used regardless of whether there is a delay or the degree of delay. Note that the X field H3 in FIG. 9 is the most important because it is used to instruct advanced operations of the SE server 140. The configuration of each field will be described in detail below with reference to FIG. 9.

[0101] The T field H1 is 1 bit in length and indicates the type of direction of traffic flow: T=0 in the T field H1 indicates the forward direction, i.e., from the leader to the follower, and T=1 indicates the reverse direction, i.e., from the follower to the leader.

[0102] The F field H2 is a syntax based on the encoder. Specifically, the F field H2 includes details of the haptic data and is composed of 12 bits when there is no delay. The F field H2 includes 3 bits for position, 3 bits for direction, 3 bits for velocity, and 3 bits for angular velocity. If there is no data related to force, the leader terminal 110 may fill the bit fields of the F field H2 with 0 (zero). Note that the leader terminal 110 generates the F field H2 based on data acquired from a sensor of the leader terminal 110 (e.g., a sensor attached to the user).

[0103] The X field H3 is an extension bit used to indicate the function to be executed by the SE server 140. The X field H3 is 3 bits long. In detail, the HAV data processing unit 113 evaluates the QoE acquired by the QoE acquisition unit 115 and determines whether the QoE satisfies the required standard. Based on the result of this determination, the X field H3 is set to enable or disable the functions of the first functional unit 141 and the second functional unit 142 of the SE server 140. A detailed flow of this process will be described later. Note that the HAV data processing unit 113 can change and set the X field H3 depending on the presence or absence of a transmission delay, or the degree of the delay, as determined in FIG. 7 .

[0104] The S field H4 is 32 bits long and contains a sequence number set by the leader terminal 110 when transmitting a packet of HAV data. Specifically, the sequence number increases by one for each sampling step. Packets transmitted from the leader terminal 110 contain the latest sequence number. In addition, the difference between the TI system timestamp and the sequence number may be used for packet drop information.

[0105] Fig. 10 shows an example of the payload structure used in the SEEHC header in Fig. 8. Fig. 10 shows the payload structure of a forward bitstream. The payload is composed of fields for position data (pos-x, pos-y, pos-z), direction data (ori-x, ori-y, ori-z), velocity data (vel-x, vel-y, vel-z), and angular velocity data (ang-x, ang-y, ang-z), each of which indicates data for three-dimensional directions. The length of each field in the payload is 32 bits.

[0106] Figure 11 shows an example of the payload structure of a feedback bitstream corresponding to the forward bitstream corresponding to Figure 10. As shown in Figure 11, the feedback bitstream is composed of force data (force-x, force-y, force-z) and torque data (torque-x, torque-y, torque-z), each of which indicates data in three-dimensional directions. The length of each field in the feedback bitstream is 32 bits.

[0107] For example, when a user of the leader terminal 110 performs an action, instructions regarding the action are transmitted from the leader terminal 110 to the follower terminal 120 as a forward bit stream as shown in Fig. 10. The follower terminal 120 controls a surgical robot connected to the follower terminal 120 based on the received instructions. The follower terminal 120 transmits data related to the force detected by a sensor provided on the robot to the leader terminal 110 as a feedback bit stream as shown in Fig. 11. Note that this feedback bit stream is transmitted directly from the follower terminal 120 to the leader terminal 110 when there is no transmission delay.

[0108] Figure 12 shows a specific example of a SEEHC header (codec header) used when the HAV data is kinesthetic data and it is determined in step S32 of Figure 7 that there is a delay. The SEEHC header includes a T field H1', ​​an F field H2', an X field H3', and an S field H4'. Below, with regard to each field in Figure 12, differences from Figure 8 will be particularly described, and descriptions of commonalities with Figure 8 will be omitted as appropriate.

[0109] The T field H1' is the same as the T field H1 in FIG. 8, and therefore a description thereof will be omitted.

[0110] The F field H2′ is an encoder-based syntax. Specifically, the F field H2′ contains details of the haptic data and is composed of 18 bits if there is a delay. The F field H2′ includes three bits for position, three bits for direction, three bits for velocity, and three bits for angular velocity. Furthermore, the F field H2′ allows six bits to be used for haptic / energy information corresponding to each dimension of the haptic data for TDPA. The haptic / energy information is information about at least one of haptic and energy. The haptic / energy information is used to control the codec in case of a delay. Furthermore, the haptic / energy information is used in the TDPA encoding process to prevent unexpected changes in data due to interference in the communication path. In this way, the energy level is minimized, ensuring system stability and passivity of the communication link. If force-related data is not present, the reader terminal 110 may fill the bit fields of the F field H2′ with zeros.

[0111] The X field H3′ is an extension bit used to indicate the function to be executed by the SE server 140. The X field H3 is 5 bits in length. Similar to the X field H3 in FIG. 8 , the HAV data processing unit 113 sets the X field H3′ to enable or disable the functions of the first functional unit 141 and the second functional unit 142 of the SE server 140 based on the QoE evaluation.

[0112] The S field H4' is 32 bits long and includes a sequence number and other information set by the leader terminal 110 when transmitting a packet of HAV data.

[0113] Figure 13 shows an example of the payload structure used in the SEEHC header in Figure 12. Figure 13 shows the payload structure of a forward bitstream. In addition to the data shown in Figure 10, the payload in Figure 13 includes additional information on translational energy (Ex, Ey, Ez) and rotational energy (Eox, Eoy, Eoz), each of which indicates data in three-dimensional directions. The payload field length is 32 bits.

[0114] Figure 14 shows an example of the payload configuration of a feedback bitstream corresponding to the forward bitstream corresponding to Figure 13. As shown in Figure 14, the feedback bitstream includes, in addition to the data shown in Figure 11, additional information regarding translational energies (Ex, Ey, Ez) and rotational energies (Eox, Eoy, Eoz), each of which indicates data in three-dimensional directions.

[0115] 13 and 14, the newly shown energy data of translational energy and rotational energy are used in TDPA, which smooths the output energy to maintain system stability and communication link passivity during remote operation when using a wide area network with delay. In this way, by adding information about energy for controlling the codec to the SEEHC header, it becomes possible to maintain system stability and communication link passivity during remote operation.

[0116] The relationship between the SEEHC and the SE server 140 will now be further described. The SEEHC takes advantage of the presence of the SE server 140 within the TI during remote operation and defines specific behaviors of the SE server 140. The combination of the SE server 140 and the SEEHC enhances the TI functionality, thereby improving and ensuring QoE in wide area networks and supporting low network latency for remote operation over distances of several hundred kilometers or more.

[0117] The SEEHC defines bits within the SEEHC frame. In particular, the bits in the X field are important as they define the operation of the SE server 140. As described above, when there is no network delay, the bit size of the X field is 3 bits, but when there is network delay, the bit size of the X field is 5 bits.

[0118] Figure 15 is a table showing actions associated with the SEEHC scheme when there is no delay. Figure 15 shows an example of an X field with a bit size of 3 bits in the SEEHC header of Figure 8. Below, we will show examples of each action shown in Figure 15.

[0119] When the X field is X11, i.e., "001," forwarding to the SE server 140 is enabled. Therefore, the leader terminal 110 transmits the HAV data that it had been transmitting to the follower terminal 120 to the SE server 140 instead of the follower terminal 120. X11 also indicates that network prediction performed by the first function unit 141 of the SE server 140 is enabled. The network prediction is, for example, a prediction regarding at least one of packet loss, transmission delay, QoS, QoE, etc. By having the first function unit 141 perform the prediction and control the HAV communication, packet loss and transmission delay can be reduced, and QoS, QoE, etc. can be improved. X11 corresponds to (A) in the first embodiment.

[0120] If the X field is X12, i.e., "011," forwarding to the SE server 140 is enabled. Therefore, the leader terminal 110 transmits the HAV data that it had previously transmitted to the follower terminal 120 to the SE server 140 instead of the follower terminal 120. X11 also indicates that the second function unit 142 of the SE server 140 is enabled to predict (expert profile) human behavior or haptic data acquired by a robot. This allows for advanced data modeling, prediction, and training. The second function unit 142 then executes prediction and controls HAV communication, thereby reducing packet loss and transmission delays and improving QoS, QoE, and the like. X11 corresponds to (B) in the first embodiment.

[0121] If the X field is X13, i.e., "101," this indicates that a procedure for detecting a new SE server 140 is being initiated. The determination of a new SE server 140 may indicate that a new SE server 140 is being determined to control communications between the leader terminal 110 and the follower terminals 120, in a state where the SE server 140 is not currently involved in the communications between the leader terminal 110 and the follower terminals 120. Alternatively, the determination of a new SE server 140 may indicate that an SE server other than the SE server currently controlling communications between the leader terminal 110 and the follower terminals 120 is being caused to control the communications. X13 is necessary, for example, when a failure in at least one of QoS or QoE occurs on either the leader side or follower side of remote operation, and another SE with higher system resources may be required.

[0122] The leader terminal 110 or the follower terminal 120 can have the new SE server 140 control communication by sending a request including X13. The first functional unit 141 or the second functional unit 142 of the SE server 140 that receives the request starts control of communication of HAV data in response to the request. In this way, the SE server that controls communication can be changed from an unselected state to a selected state, or the SE server that controls communication can be changed to a state where an SE server that can further improve QoE, etc. is selected, thereby making it possible to improve communication performance of HAV data.

[0123] If the X field is X14, i.e., "111," it indicates that the teardown sequence is about to begin, which is necessary to indicate to the SE server 140 that the remote control session has ended and that any SE server 140 resources that were being used can be released.

[0124] The leader terminal 110 or the follower terminal 120 can release resources of the SE server 140 that was being used by sending a trigger including X14 to the SE server 140. The first functional unit 141 or the second functional unit 142 of the SE server 140 releases the resources in response to the trigger. In this way, the SE server 140 becomes capable of controlling other communications, thereby improving communication performance not only between the leader terminal 110 and the follower terminal 120 but also throughout the entire network.

[0125] For example, in a remote operation scenario where multimodal data (HAV data) is transmitted and received over a network over short distances, such as within the same room or building, there may be no issues related to transmission delay. However, issues such as jitter, out-of-order data packets, interference, motion force control, impedance control, communication link passivity, system stability, and system drift between leaders and followers may be issues that need to be addressed even in a latency-free network. However, even in a latency-free network, the use of the SEEHC described above for data transmission and reception enables advanced methods to address issues other than latency, and can take preventative measures against transmission link failures. As a result, the network can provide a higher QoE.

[0126] Figure 16 is a table showing actions related to the SEEHC scheme when there is a delay. Figure 16 shows an example of an X field with a bit size of 5 bits in the SEEHC header of Figure 12. When the X field in Figure 16 is X21, i.e., "00001", it corresponds to when the X field in Figure 15 is X11, i.e., "001". When the X field in Figure 16 is X22, i.e., "00011", it corresponds to when the X field in Figure 15 is X12, i.e., "011". When the X field in Figure 16 is X23, i.e., "10001", it corresponds to when the X field in Figure 15 is X13, i.e., "101". When the X field in Figure 16 is X24, i.e., "11111", it corresponds to when the X field in Figure 15 is X14, i.e., "111".

[0127] For remote operation in the presence of delays, the SE server 140 needs to enable additional functions to support a high level of QoE. Specifically, the SE server 140 needs to be instructed to perform specific functions to reduce transmission delays, increase the QoE level, and maintain system stability (passivity of the communication link). In such cases, the leader terminal 110 identifies the SEEHC by using the X field shown in Figure 16.

[0128] Next, the procedure for detecting the SE server of each of the leader terminal and the follower terminal will be described below.

[0129] FIG. 17 is a sequence diagram showing an SE server detection procedure initiated and executed by a leader terminal. FIG. 17 illustrates a wide-area remote control system including a leader-side physical / virtual endpoint D1, a leader terminal D2, an SE server (1) D3, an SE server (n) D4, a follower terminal D5, and a follower-side physical / virtual endpoint D6. The leader terminal D2, the follower terminal D5, the SE server (1) D3, and the SE server (n) D4 correspond to the leader terminal 110, the follower terminal 120, and the SE server 140 in FIG. 5, respectively. Furthermore, the physical / virtual endpoint D1 corresponds to at least one of the physical environment 111 or the virtual environment 112 in FIG. 5, and the physical / virtual endpoint D6 corresponds to at least one of the physical environment 121 or the virtual environment 122 in FIG. 5. However, the physical / virtual endpoints D1 and D2 may be installed in the leader terminal D2 and the follower terminal D5, respectively. Alternatively, the physical / virtual endpoints D1 and D2 may be separate devices from the leader terminal D2 and the follower terminal D5, respectively, and may be devices connected to the leader terminal D2 and the follower terminal D5. Furthermore, the remote control system is equipped with multiple SE servers, numbering n. The detection procedure will be described below with reference to FIG. 17.

[0130] The physical / virtual endpoint D1 transmits data related to the remote operation to the leader terminal D2 (step S51). The leader terminal D2 transfers the received data to the follower terminal D5 (step S52). The follower terminal D5 transfers the received data to the physical / virtual endpoint D6 that is the target of the remote operation session (step S53). By the operations up to this point, the data related to the remote operation is transmitted to the physical / virtual endpoint D6, and the remote operation instructed by the user of the physical / virtual endpoint D1 is performed on the physical / virtual endpoint D6 side.

[0131] Next, the physical / virtual endpoint D1 transmits the QoE performance (hereinafter also referred to as QoE(L)) on the leader side to the leader terminal D2 (step S54). In this example, the physical / virtual endpoint D1 notifies the leader terminal D2 that the QoE(L) does not meet the required standard. This notification is made, for example, by the user performing the remote operation operating the input unit of the physical / virtual endpoint D1.

[0132] In response to the received notification, the leader terminal D2 starts detecting an SE server and transmits an indicator notifying the start of SE server detection to the physical / virtual endpoint D1 (step S55). By transmitting this indicator, for example, processing such as stopping further data transmission from the physical / virtual endpoint D1 is performed until the SE server is detected.

[0133] Then, the leader terminal D2 starts the SE server detection process (step S56). The leader terminal D2 sends an SE server request (hereinafter also referred to as SE request) to all SE servers to which the leader terminal D2 can connect, including SE server (1) D3 and SE server (n) D4 (steps S57 and S58). This SE request has an SEEHC header including an X13 or X23 field, which indicates that a new SE server detection procedure is being initiated. The SE request is sent to candidates for SE servers that will execute communication control. SE server (1) D3 and SE server (n) D4 receive the SE request and send a response (hereinafter also referred to as SE response) to the leader terminal D2 (steps S59 and S60).

[0134] Based on all information necessary to select the optimal SE server, the leader terminal D2 selects an optimal SE server from among the SE servers that received the SE response, replacing the currently selected SE server (step S61). The leader terminal D2 may select the SE server located closest to the leader terminal D2 on the communication path. However, the closest SE server may be busy or under heavy load. In that case, another SE server that can provide the necessary resources, such as system load, HAV data latency requirements, and communication link stability, may be selected. The leader terminal D2 identifies an SE server that can provide the necessary resources based on the content of the received SE response, and selects the identified SE server.

[0135] When the leader terminal D2 selects the SE server (1) D3 as the optimal SE server, the leader terminal D2 transmits a resource reservation request (hereinafter also referred to as an SE reservation) to the selected SE server (1) D3 (step S62). In response to receiving the SE reservation from the leader terminal D2, the SE server (1) D3 checks whether there are available resources in its own device. Here, the SE server (1) D3 confirms that there are available resources and transmits an SE response indicating that there are available resources to the leader terminal D2 (step S63).

[0136] In response to receiving the SE response from the SE server (1) D3, the leader terminal D2 notifies the follower terminal D5 of the selection of the SE server (1) D3 as the new SE server and transmits an SE handover request to the follower terminal D5 to execute a handover of the SE server (step S64). In response to receiving the SE handover request, the follower terminal D5 transmits an SE reservation to the SE server (1) D3 indicated in the SE handover request (step S65). In response to receiving the SE reservation from the follower terminal D5, the SE server (1) D3 transmits an SE response to the follower terminal D5 (step S66).

[0137] The follower terminal D5, which has received the SE response, transmits a handover response to the leader terminal D2 indicating that handover of the SE server is possible (step S67). The follower terminal D5 also notifies the physical / virtual endpoint D6 of an indicator indicating that the SE server to be used will be switched to SE server (1) D3 (selection result) (step S68). Similarly, the leader terminal D2, which has received the handover response, recognizes that the SE server to be used can be switched to SE server (1) D3. The leader terminal D2 then notifies the physical / virtual endpoint D1 of an indicator indicating that the SE server to be used will be switched to SE server (1) D3 (step S69).

[0138] After the SE server being used is switched to the SE server (1) D3, the physical / virtual endpoint D1 resumes sending data related to the remote operation to the leader terminal D2 (step S70). The leader terminal D2 transfers the received data to the selected SE server (1) D3 (step S71). The SE server (1) D3 analyzes the received data using a model held by the SE server and modifies the data. Details of this analysis and modification process are as described above. The SE server (1) D3 transmits the modified data to the follower terminal D5 (step S72). The follower terminal D5 transfers the received data to the physical / virtual endpoint D6 that is the target of the remote operation session (step S73). Through the operations up to this point, the data related to the remote operation is modified and then transmitted to the physical / virtual endpoint D6. On the physical / virtual endpoint D6 side, remote operation is performed as instructed by the user of the physical / virtual endpoint D1 and adjusted to suit the remote operation.

[0139] Next, the physical / virtual endpoint D1 transmits the QoE(L) to the leader terminal D2 (step S74). In this example, the physical / virtual endpoint D1 notifies the leader terminal D2 that the QoE(L) satisfies the required criteria. Therefore, a new SE server is not selected, and the SE server selection performed up to step S69 is completed. In other words, remote operation between the physical / virtual endpoints D1 to D6 continues using the settings performed up to step S69. If the QoE(L) no longer satisfies the required criteria, the SE server selection process similar to that shown in steps S54 to S69 is restarted. At this time, the leader terminal D2 may transmit to the SE server (1) D3 an SEEHC header including a field X14 or X24 instructing the SE server (1) D3 to release its resources as a trigger for releasing the resources.

[0140] The processes shown in steps S54 to S69 may be executed not only when a new SE server is selected, but also when at least one of the first functional unit 141 and the second functional unit 142 in an already selected SE server is enabled, disabled, or set to ON or OFF. In other words, these processes may be executed when there is any change in the method for controlling HAV communication based on network parameters or at least one prediction regarding humans or robots.

[0141] Fig. 18 is a sequence diagram showing a procedure for detecting an SE server that is initiated and executed by a follower terminal. The components of the remote control system shown in Fig. 18 are the same as those shown in Fig. 17. The detection procedure will be described below with reference to Fig. 18.

[0142] The physical / virtual endpoint D1 transmits data related to the remote operation to the leader terminal D2 (step S81). The leader terminal D2 transfers the received data to the follower terminal D5 (step S82). The follower terminal D5 transfers the received data to the physical / virtual endpoint D6 that is the target of the remote operation session (step S83). By the operations up to this point, the data related to the remote operation is transmitted to the physical / virtual endpoint D6, and the remote operation instructed by the user of the physical / virtual endpoint D1 is performed on the physical / virtual endpoint D6 side.

[0143] Next, the physical / virtual endpoint D6 transmits the QoE performance (hereinafter also referred to as QoE(F)) on the follower side to the follower terminal D5 (step S84). In this example, the physical / virtual endpoint D6 notifies the follower terminal D5 that the QoE(F) does not meet the required standard. This notification is made, for example, by a sensor on the follower side robot that is performing the instructed remote operation on-site detecting the QoE(F). Alternatively, this notification may be made by a user on the follower terminal D5 operating an input unit of the physical / virtual endpoint D6.

[0144] In response to the received notification, the follower terminal D5 starts detecting an SE server and transmits an indicator notifying the start of SE server detection to the physical / virtual endpoint D6 (step S85). By transmitting this indicator, for example, processing such as stopping further data transmission from the physical / virtual endpoint D6 until the SE server is detected is performed.

[0145] Then, the follower terminal D5 starts the SE server detection process (step S86). The follower terminal D5 sends an SE request to all SE servers to which the follower terminal D5 can connect, including SE server (1) D3 and SE server (n) D4 (steps S87 and S88). This SE request may have an SEEHC header including an X13 or X23 field, which indicates that a new SE server detection procedure is being started. The SE server (1) D3 and SE server (n) D4 receive the SE request and send an SE response in response to it to the follower terminal D5 (steps S89 and S90).

[0146] Based on all the information necessary to select the optimal SE server, the follower terminal D5 selects the optimal SE server from among the SE servers from which the SE response was received, replacing the currently selected SE server (step S91). Details of how the follower terminal D5 selects the SE server are as described in step S61. When the follower terminal D5 selects the SE server (1) D3 as the optimal SE server, the follower terminal D5 transmits an SE reservation to reserve resources to the SE server (1) D3 to be selected (step S92). In response to receiving the SE reservation from the follower terminal D5, the SE server (1) D3 checks whether there are available resources in its own device. Here, the SE server (1) D3 confirms that there are available resources and transmits an SE response to the follower terminal D5 indicating that there are available resources (step S93).

[0147] In response to receiving the SE response from the SE server (1) D3, the follower terminal D5 notifies the leader terminal D2 that the SE server (1) D3 has been selected as the new SE server and transmits an SE handover request to execute a handover of the SE server (step S94). In response to receiving the SE handover request, the leader terminal D2 transmits an SE reservation to the SE server (1) D3 indicated in the SE handover request (step S95). In response to receiving the SE reservation from the leader terminal D2, the SE server (1) D3 transmits an SE response to the leader terminal D2 (step S96).

[0148] Upon receiving the SE response, the leader terminal D2 transmits a handover response to the follower terminal D5 indicating that handover of the SE server is possible (step S97). The leader terminal D2 also notifies the physical / virtual endpoint D1 of an indicator indicating that the SE server to be used will be switched to SE server (1) D3 (selection result) (step S98). Similarly, upon receiving the handover response, the follower terminal D5 recognizes that it is possible to switch the SE server to be used to SE server (1) D3. The follower terminal D5 then notifies the physical / virtual endpoint D6 of an indicator indicating that the SE server to be used will be switched to SE server (1) D3 (step S99).

[0149] After the SE server being used is switched to the SE server (1) D3, the physical / virtual endpoint D1 resumes sending data related to the remote operation to the leader terminal D2 (step S100). The leader terminal D2 transfers the received data to the selected SE server (1) D3 (step S101). The SE server (1) D3 analyzes the received data using a model held by the SE server and modifies the data. Details of this analysis and modification process are as described above. The SE server (1) D3 transmits the modified data to the follower terminal D5 (step S102). The follower terminal D5 transfers the received data to the physical / virtual endpoint D6 that is the target of the remote operation session (step S103). Through the operations up to this point, the data related to the remote operation is modified and then transmitted to the physical / virtual endpoint D6. On the physical / virtual endpoint D6 side, remote operation is performed as instructed by the user of the physical / virtual endpoint D1 and adjusted to suit the remote operation.

[0150] Next, the physical / virtual endpoint D6 transmits the QoE (F) to the follower terminal D5 (step S104). In this example, the physical / virtual endpoint D6 notifies the follower terminal D5 that the QoE (F) satisfies the required criteria. Therefore, a new SE server is not selected, and the SE server selection performed up to step S99 is completed. In other words, remote operation between the physical / virtual endpoints D1 to D6 continues using the settings performed up to step S69. If the QoE (F) no longer satisfies the required criteria, the SE server selection process similar to that shown in steps S84 to S99 is restarted. At this time, the follower terminal D5 may transmit to the SE server (1) D3 an SEEHC header including a field X14 or X24 instructing the SE server (1) D3 to release its resources as a trigger for releasing the resources.

[0151] The processes shown in steps S84 to S99 may be executed not only when a new SE server is selected, but also when at least one of the first functional unit 141 and the second functional unit 142 in an already selected SE server is enabled, disabled, or set to ON or OFF. In other words, these processes may be executed when any change occurs in the method for controlling HAV communication based on network parameters or predictions regarding humans or robots.

[0152] Next, we will explain the training of the SE server 140. The SE-EPS (SE Expert Profile Selection) scheme is used for training the SE server 140. The SE-EPS scheme can use expert profiles and templates initiated by the SE server 140 to train the SE server 140's second model 144 to further optimize QoE during teleoperation. The SE-EPS scheme supports haptic prediction with a transmission delay of less than 1 ms when the prediction error is minimized. The SE-EPS scheme is a learning-by-demonstration approach in which a haptic-sensing robot operated by teleoperation at the follower terminal 120 acquires skills without being explicitly programmed. A hidden Markov model (HMM) is used for offline training by encoding force and torque data and related parameters. Once the HMM is trained, a generalized profile can be extracted. The HMM is useful for predicting forward and feedback information that may be lost during data transmission over a wide area network.

[0153] FIG. 19 is a diagram illustrating the training of the SE server 140 using expert profiles. As shown in FIG. 19, the HMM may be trained by multiple experts. When different objects are used in the physical / virtual world E1, such as devices for telemedicine, objects handled in logistics, or moving objects such as drones or vehicles, the expert E2 uses such objects to create an expert profile E3. The expert E2 then uses the expert profile E3 to train the SE of the SE server E4. This turns the SE into a trained agent, improving the accuracy of the model. The trained SE server E5 then makes predictions regarding the network and / or humans and / or robots, thereby contributing to reducing transmission failures and delays.

[0154] It should be noted that multiple objects E11 may exist in the physical / virtual world E1, and each object E11 may be different. If each object E11 is different, a dedicated trainer E12 is required for training each object E11. As a result of training by the trainer E12, a different profile E13 is created for each object E11. For example, a surgeon can use a dedicated trainer E12 to generate a teleoperation profile E13. A data profile related to the teleoperation is also generated. At each step of the training session, position, direction, force, and torque information is generated, and a set of data profiles is encoded using an HMM. The encoded data profiles are then sent to SE E14. The trained SE E15 then reproduces the generalized model during the online session.

[0155] The profile E13 described above may be created outside the SE server 140 (for example, in the leader terminal 110 or the follower terminal 120). The created profile E13 is input to the SE server E4, thereby training the SE of the SE server E4. This training is performed, for example, when the SE server 140 is in passive mode (i.e., when at least one of the first model 143 and the second model 144 is not used). However, when the SE server 140 is in active mode, i.e., when at least one of the first model 143 and the second model 144 is used, the above training may be performed as online training.

[0156] The HMM is used to encode a set of force, torque, and velocity data into a set of profiles. After training the HMM, Gaussian Mixture Regression (GMR) can be used by the SE server E4 to support online teleoperation sessions. Using HMM / GMR, the SE server E4 can provide predicted data to the follower terminal 120 if data from the leader terminal 110 is lost or delayed. In this scenario, the transmission delay can be kept below 1 ms, allowing the lost data to be restored. This HMM / GMR model can be used to predict teleoperation haptic feedback information, achieving a low error rate over a wide area network with a prediction time of less than 1 ms.

[0157] [Effectiveness] The evolution of wide area networks has made it easy to connect people and devices that are far apart via wide area networks, such as between municipalities or countries. Furthermore, most electronic devices, sensors, and actuators can provide easily accessible general-purpose APIs. Combining wide area networks and device control via APIs has brought about new use cases such as remote steering and remote medical care.

[0158] However, such remote control of robots, especially those involving the transmission of multimodal data, is a challenging task for current wide-area networks due to data loss and delays. Multimodal data includes, for example, HAV data. Wide-area networks can adversely affect user experience due to long wired and wireless connection links. User experience is highly important and is measured as QoE. Current remote control systems have limited application ranges and provide insufficient QoE over wide-area networks. Therefore, remote control is primarily performed over short distances, such as within the same room or building, but is difficult to achieve over long distances, such as between municipalities or countries. When performing such long-distance remote control, users may face challenges such as delayed or lost HAV data, which can lead to network unreliability due to poor QoE.

[0159] Therefore, it is important that the communication link between the leader and the follower does not affect the remote control and can support the stability of the system. For example, even when remote control is performed over a long distance, such as between different municipalities or countries, it is desirable that the QoE be indistinguishable or nearly indistinguishable from that of local remote control performed within the same room or building.

[0160] Furthermore, long-distance remote control has very important requirements for the communication path. The communication path (e.g., a path for a wireless connection such as 5G or 6G) should not interfere with the remote control itself. The remote control endpoint (e.g., a leader terminal or a follower terminal) is a highly complex system including sensors, actuators, motors, springs, and dampers. Therefore, data transmission over the communication path may affect the stability of not only the follower side but also the leader side. For this reason, the remote control system must guarantee the system stability of the entire system. In particular, it is most important that the communication path has low latency, behaves passively, and reduces system drift between the leader and follower. However, guaranteeing this system stability is a difficult requirement to achieve over a wide-area communication network. Specifically, data transmission over a wide-area network such as 5G or 6G may cause unwanted energy increases or changes in system impedance.

[0161] Haptic data communication requires network latency of approximately 1 ms to properly control a physical machine and a virtual environment from a remote location. However, in wide-area networks such as 5G or 6G, technical and physical constraints make it difficult to achieve network latency of 1 ms or less. The technical constraints include not only latency in the wireless interface but also latency in the wired backbone network. The physical constraints are due to the limited speed of light, which causes latency in wide-area networks and limits the distance over which the network can be used.

[0162] Specifically, haptic data includes kinesthetic data and / or tactile data. Kinesthetic data includes data related to force, torque, and pressure. These data are highly susceptible to transmission delays, jitter, and packet loss. Delays or losses in data signals can result in events that are not perceptible to human perception. Furthermore, delays in kinesthetic data can lead to incorrect or delayed decisions by human perception, i.e., the human brain. Even if a human perceives kinesthetic data, which is an important signal, it may be ignored if the data arrives late.

[0163] Furthermore, haptic data is data that allows humans to sense the surface of an object through their senses. Such data is less susceptible to transmission delays. However, when combining audio and video data with haptic data, synchronization of the haptic data, audio data, and video data is important to achieve a high level of QoE. Therefore, it is important that HAV data transmission be supported with high network bandwidth, low latency, and low jitter time.

[0164] In light of the above background, the present inventors have discovered that adding network components and advanced algorithms to a network, as described in this disclosure, can improve network speed and reduce latency. The remote control system 100 according to the second embodiment can reduce the transmission latency of HAV data over a wide area network, such as 5G (e.g., 5G Advanced) or 6G, with a distance of, for example, 100 km or more. Therefore, it is possible to support haptic applications such as remote control.

[0165] In detail, the above effect is achieved by having the network of the remote operation system 100 switch data traffic between the leader terminal and the follower terminals, thereby supplying data to an SE that provides a function of improving QoE for remote users. In addition to the functions of a receiver and a transmitter, the SE has an internal module (first function unit 141) that predicts network parameters and an internal module (second function unit 142) that predicts the user or robot.

[0166] The SE can improve traffic quality and make the system in which remote control is realized more stable by predicting network parameters using the first functional unit 141. Furthermore, the SE can predict human behavior using the second functional unit 142, and instantly feed back information about the predicted user behavior or haptic data acquired by the robot to the remote control system before new system data is transmitted to the SE. Therefore, the SE can improve QoE on a wide area network without being affected by network delays.

[0167] Here, identifying QoE parameters on both the leader and follower sides is important to improve QoE as needed. Identifying communication constraints within the TI can improve QoE. Such communication constraints may be based on network parameters such as transmission delay, jitter, and packet loss. The terminal in the second embodiment can identify whether an SE server exists and whether such SE server can support a remote control session. The SE server then supports the remote control session by receiving HAV data from the leader terminal and forwarding the HAV data to the follower terminal. The SE server can also support the remote control session by receiving HAV data from the follower terminal and forwarding the HAV data to the leader terminal.

[0168] The SE server has the following modes: (1) The SE server can be in a passive mode in which it does not change the HAV data received from the leader terminal and transfers the HAV data from the leader terminal to the follower terminals. In this passive mode, the second function unit 142 of the SE server is disabled, so the SE server does not make predictions about users or robots.

[0169] (2) When the SE server receives HAV data from the leader terminal, it can be in an active mode in which it modifies the HAV data and transfers the modified data to the follower terminal. In this active mode, the second function unit 142 of the SE server is enabled, so the SE server makes predictions about the user or robot and modifies the received HAV data based on the prediction results.

[0170] In this way, by selecting the SE to be used, it is possible to extend the communication range of the TI beyond physical boundaries and also improve the QoE of the TI during remote operation.

[0171] The SE server can also apply additional functions and techniques, such as prediction, to the data being transmitted and received. This prediction allows the SE to improve QoE performance. The SE server can use trained expert models, network prediction, or additional encoding schemes to predict human behavior or robot-acquired haptic data, further reducing network latency or improving overall QoE performance. For example, in the second embodiment, the SE server can perform training using the SE-EPS to obtain an advanced prediction model.

[0172] In either the passive mode or the active mode, the first functional unit 141 of the SE server 140 may be enabled or disabled. When the first functional unit 141 is enabled, the SE server 140 can predict network parameters to improve data transmission performance (or the accuracy of network prediction). In this way, the first functional unit 141 and the second functional unit 142 of the SE server 140 can be individually set to enable or disable their functions as needed. To improve QoE, either the first functional unit 141 or the second functional unit 142 of the SE server 140 may be set to enable, or both may be set to enable.

[0173] Additionally, multiple SEs, each with different capabilities, such as prediction, remote rendering, or computation outsourcing, may be provided in the network. Each SE may provide a unique set of capabilities to be used during a remote operation session. As described above, the second embodiment describes a specific SE selection procedure for selecting the most appropriate SE. This SE selection is part of an encoding scheme for indicating the internal capabilities of the selected SE server 140. The SEEHC instructs the SEs regarding prediction capabilities and provides data fields for configuring capabilities at the SE, leader, and follower sides. In this case, devices with low power consumption, such as remote rendering or computation outsourcing, can be provided in the network as SEs. Therefore, even when the network's capabilities are expanded, the increase in power consumption required for the network can be suppressed.

[0174] Fig. 20 is a diagram illustrating the effects of the present disclosure. In Fig. 20, the leader terminal 110 is connected to the follower terminal 120 via the SE server 140. The configurations of the leader terminal 110 and the follower terminal 120 are the same as those shown in Fig. 5, and therefore description thereof will be omitted. The SE server 140 also includes a rendering / modeling unit 14A, a prediction unit 14B, and an interface 14C. The rendering / modeling unit 14A is a functional unit that collectively refers to the first functional unit 141 and the second functional unit 142, and the prediction unit 14B is a functional unit that collectively refers to the first model 143 and the second model 144. The interface 14C corresponds to the interface 145.

[0175] The leader terminal 110 has a coverage area (hereinafter also referred to as area A) in which data transmission from the leader terminal 110 is possible, determined by a data connection link LA (hereinafter also referred to as link LA). When link LA crosses horizon HA, which is the boundary of area A, link A becomes unstable, lossy, or delay-prone. In the area beyond horizon HA from the leader terminal 110, it is not practical to use the QoE(L) of the leader terminal 110 for the QoE adjustment function, which is a function of link LA. In Figure 20, event F1 indicates that QoE(L) cannot be used for the QoE adjustment function in the area beyond horizon HA.

[0176] Similarly, the follower terminal 120 has a coverage area (hereinafter also referred to as area B) in which data transmission from the follower terminal 120 is possible, which is determined by a data connection link LB (hereinafter also referred to as link LB). When link LB exceeds horizon HB, which is the boundary of area B, link B becomes unstable, lossy, or experiences delays. In the area beyond horizon HB from the follower terminal 120, it is not practical to use the QoE (F) of the follower terminal 120 for the QoE adjustment function, which is a function of link LB. In FIG. 20, the fact that QoE (F) cannot be used for the QoE adjustment function in the area beyond horizon HB is shown as event F2.

[0177] Here, when link LA exceeds horizon HA, the leader terminal 110 acquires not only QoE(L) but also QoE(F) at the follower terminal 120 and detects the difference between QoE(L) and QoE(F). For example, if the detection result indicates a network failure, the leader terminal 110 initiates negotiation with the SE server 140 to select a new SE server 140. When link LB exceeds horizon HB, the follower terminal 120 can perform the same process instead of the leader terminal 110. In this way, both the leader terminal 110 and the follower terminal 120 can make decisions about network performance. This enables predictive models, outsourced rendering, or modeling.

[0178] The SE Server 140 supports teleoperations over wide area networks by predicting network performance metrics and human activity, thereby guaranteeing required latency parameters and enabling life-critical operations such as remote surgery. This enables stable and reliable TI for HAV application scenarios, as well as high-reliability and low-latency communications for smart devices, industrial Internet-of-Things (IoT), and telemedicine.

[0179] The above-described embodiment can be modified, for example, as follows. The present disclosure may be fully embodied as a V2V (Virtual to Virtual) environment in which a communication link including an SE forms a network. However, human behavior and the acquisition of that behavior data may be performed in a virtual space using Google Glass (registered trademark), HoloLens (registered trademark), or the like. The leader terminal transmits data related to the behavior performed in the virtual space to the follower terminal, and at that time, the SE server can predict the human behavior. Therefore, the SE server can provide data at any time when lost data occurs.

[0180] The SE server can also fully render an entire session based only on basic data input from the leader terminal. This is interesting for multi-game operations because the graphics-intensive and costly rendering can be managed by the SE server. For example, finite element simulations and 3D (Three Dimensions) rendering require high-performance graphical processing units (GPUs). GPUs can be stored in large quantities on the SE server, similar to a so-called rendering farm, enabling the SE server to perform rendering in multi-game operations, etc. Therefore, the system disclosed herein can be applied to outsourced graphical rendering applications, including AI graphical rendering models, and can be monetized.

[0181] The following is a summary of applications of the present disclosure. The servers and terminals described in each embodiment of the present disclosure are applicable to, but not limited to, telemedicine, telesurgery, telerobotic surgery, and robotic surgery for the purposes of, for example, biopsy, injection, neurosurgery, or cancer treatment. The present disclosure is used, for example, for emulation between a human and a machine at a distance using a reinforcement learning unit supervised for teleoperation.

[0182] Ultra-low latency wide area network support enables secure remote operation between distant locations over wide area communication networks for applications such as proactive healthcare, industrial automation, intelligent transportation, and surveillance for defense and disaster management.

[0183] Remote communication according to the present disclosure enables numerous edutainment applications and immersive human experiences via remote machines, including immersive virtual / augmented reality (VR / AR), and enables immersive control and manipulation of real and virtual objects and machines.

[0184] An AI-enhanced server on the access network enables prediction of haptic manipulation, including offline expert training and online simulation of haptic manipulation through haptic recognition, kinesthetic recognition, or tactile recognition.

[0185] The present disclosure can support remote master-replica pair control, including sensing and control of real and virtual machines and robots, as well as predictive HAV data. The present disclosure can also support haptic internet applications, including smart factories and smart logistics.

[0186] At least one of the terminals and servers shown in each embodiment is realized by one or more computers. When each part of the terminal or server is realized by multiple computers, the method of distributing each part of the terminal or server among the multiple computers is arbitrary.

[0187] Some or all of the components of the terminal or server may be provided in a cloud server built on the cloud, or in other types of virtualized servers created using virtualization technology, etc. Functions other than those provided in servers such as cloud servers or virtualized servers are placed at the edge. For example, in a wide area network that realizes remote surgery, the edge is a device placed at or near the site of surgery.

[0188] In the above-described embodiment, this disclosure has been described as a hardware configuration, but this disclosure is not limited to this. In this disclosure, the processes or steps performed by at least any of the terminals, management servers, and SE servers described in the above-described embodiment may be realized by causing a processor in a computer to execute a computer program.

[0189] 21 is a block diagram showing an example of the hardware configuration of an information processing device 90 that executes the processes of the devices in each of the above-described embodiments. Referring to FIG. 21, this information processing device 90 includes a signal processing circuit 91, a processor 92, and a memory 93.

[0190] The signal processing circuit 91 is a circuit for processing signals in accordance with the control of the processor 92. The signal processing circuit 91 may include a communication circuit for receiving signals from a transmitting device.

[0191] The processor 92 is connected or coupled to the memory 93, and performs the processing of the device described in the above embodiment by reading and executing software or computer programs from the memory 93. Examples of the processor 92 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Demand-Side Platform), and an ASIC (Application Specific Integrated Circuit). A single processor may be used as the processor 92, or multiple processors may be used in cooperation with each other.

[0192] The memory 93 may be a volatile memory, a nonvolatile memory, or a combination thereof. The volatile memory may be, for example, a random access memory (RAM) such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The nonvolatile memory may be, for example, a read only memory (ROM) such as a programmable read only memory (PROM) or an erasable programmable read only memory (EPROM), a flash memory, or a solid state drive (SSD). The memory 93 may be a single memory or a combination of multiple memories.

[0193] The memory 93 is used to store one or more instructions. Here, the one or more instructions are stored as a group of software modules in the memory 93. The processor 92 can perform the processes described in the above embodiments by reading and executing the group of software modules from the memory 93.

[0194] The memory 93 may include a memory provided outside the processor 92, as well as a memory built into the processor 92. The memory 93 may also include a storage device located away from the processors constituting the processor 92. In this case, the processor 92 can access the memory 93 via an I / O (Input / Output) interface.

[0195] As described above, one or more processors included in each device in the above-described embodiments execute one or more programs including instructions for causing a computer to execute the algorithms described using the drawings. This processing enables the information processing described in each embodiment to be realized.

[0196] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disk (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0197] The above-described embodiments are merely examples of application of the technical ideas obtained by the inventors of the present invention. In other words, the technical ideas are not limited to the above-described embodiments, and various modifications are possible.

[0198] For example, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. Some or all of the elements described in any supplementary note may be applied to various hardware, software, recording means for recording software, systems, and methods. (Supplementary Note 1) A terminal comprising: a receiving unit that receives haptic data from another terminal connected via a network; a generating unit that generates, based on an indicator indicating a communication status of the haptic data, instruction information addressed to a management server on the network for executing control regarding communication of the haptic data with the other terminal; and a transmitting unit that transmits the instruction information to the management server. (Supplementary Note 2) The terminal described in Supplementary Note 1, wherein the generating unit predicts parameters related to a network between the terminal and the other terminal based on the indicator, and generates, based on the predicted parameters, the instruction information addressed to the management server for controlling communication between the terminal and the other terminal. (Supplementary Note 3) The terminal according to Supplementary Note 1 or 2, wherein the generation unit predicts haptic data to be generated on the other terminal side based on the index, and generates the instruction information addressed to the management server for controlling the management server to change the content of the haptic data to be transmitted to the terminal based on the predicted haptic data. (Supplementary Note 4) The terminal according to any one of Supplements 1 to 3, wherein the generation unit generates a request to newly detect the management server that executes control regarding communication of the haptic data with the other terminal based on the index, and the transmission unit transmits the request to candidate management servers to be changed. (Supplementary Note 5) The terminal according to any one of Supplements 1 to 4, wherein the generation unit generates a trigger to release resources of the management server, and the transmission unit transmits the trigger to the management server. (Supplementary Note 6) The terminal according to any one of Supplementary Notes 1 to 5, further comprising: a change unit that determines whether the haptic data is kinesthetic data or tactile data, and changes an encoder used to communicate the haptic data based on a determination result.(Supplementary Note 7) The terminal according to Supplementary Note 6, wherein, if the haptic data is kinesthetic data, the change unit further determines a degree of delay occurring in a network used for the communication, and changes an encoder used for communicating the haptic data based on the determined degree of delay. (Supplementary Note 8) The terminal according to Supplementary Note 6 or 7, wherein, if the haptic data is kinesthetic data, the change unit further determines whether or not a delay occurs in a network used for the communication, and, if the delay exists, adds information about energy for controlling a codec to a header of the haptic data compared to a case where the delay does not exist, and transmits the header to the other terminal. (Supplementary Note 9) A management server comprising: a receiving unit connected to a first terminal via a network, and configured to receive, from a second terminal that receives haptic data from the first terminal, instruction information for executing control related to communication of the haptic data with the first terminal; and a control executing unit that executes control related to the communication of the haptic data based on the instruction information. (Supplementary Note 10) The management server according to Supplementary Note 9, wherein the control execution unit predicts parameters related to a network between the first terminal and the second terminal based on the instruction information, and controls communication between the first terminal and the second terminal based on the predicted parameters. (Supplementary Note 11) The management server according to Supplementary Note 9 or 10, wherein the control execution unit predicts haptic data to be generated on the first terminal based on the instruction information, and controls to change content of the haptic data to be transmitted to the second terminal based on the predicted haptic data. (Supplementary Note 12) The management server according to any one of Supplements 9 to 11, wherein the receiving unit receives a request from the second terminal to execute control related to communication of the haptic data with the first terminal, and the control execution unit starts control related to communication of the haptic data in response to the request.(Supplementary Note 13) The management server according to any one of Supplementary Notes 9 to 12, wherein the receiving unit receives a trigger from the second terminal to release resources of the management server, and the control execution unit releases resources of the management server in response to the trigger. (Supplementary Note 14) A communication system comprising: a terminal; and a management server, wherein the terminal has: a receiving unit that receives haptic data from another terminal connected via a network, a generating unit that generates, based on an indicator of a communication state of the haptic data, instruction information for the management server on the network to execute control regarding communication of the haptic data with the other terminal, and a transmitting unit that transmits the instruction information to the management server, and the management server has: a receiving unit that receives the instruction information from the terminal, and a control execution unit that executes control regarding communication of the haptic data based on the instruction information. (Supplementary Note 15) The communication system according to Supplementary Note 14, wherein the generation unit predicts parameters related to a network between the terminal and the other terminal based on the index, and generates the instruction information addressed to the management server for controlling communication between the terminal and the other terminal based on the predicted parameters, and the control execution unit predicts parameters related to a network between the terminal and the other terminal based on the instruction information, and controls communication between the terminal and the other terminal based on the predicted parameters. (Supplementary Note 16) The communication system according to Supplementary Note 14 or 15, wherein the generation unit predicts haptic data to be generated on the other terminal based on the index, and generates the instruction information addressed to the management server for controlling the management server to change content of the haptic data to be transmitted to the terminal based on the predicted haptic data, and the control execution unit predicts haptic data to be generated on the other terminal based on the instruction information, and controls the management server to change content of the haptic data to be transmitted to the terminal based on the predicted haptic data.(Supplementary Note 17) A method executed by a computer: receiving haptic data from another terminal connected via a network; generating instruction information addressed to a management server on the network for executing control over communication of the haptic data with the other terminal based on an indicator indicating a communication status of the haptic data; and transmitting the instruction information to the management server. (Supplementary Note 18) A method executed by a computer: receiving instruction information from a second terminal connected to a first terminal via a network and receiving haptic data from the first terminal for executing control over communication of the haptic data with the first terminal; and executing control over communication of the haptic data based on the instruction information. (Supplementary Note 19) A program that causes a computer to execute the following steps: receiving haptic data from another terminal connected via a network; generating instruction information addressed to a management server on the network for executing control over communication of the haptic data with the other terminal based on an indicator indicating a communication status of the haptic data; and transmitting the instruction information to the management server. (Supplementary Note 20) A program that causes a computer to execute the following: receiving instruction information from a second terminal that is connected to a first terminal via a network and receives haptic data from the first terminal, for executing control regarding communication of the haptic data with the first terminal; and executing control regarding the communication of the haptic data based on the instruction information.

[0199] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, all or any part of the contents of each embodiment can be combined with other embodiments as appropriate.

[0200] This application claims priority based on Japanese Patent Application No. 2023-143543, filed September 5, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0201] REFERENCE SIGNS LIST 10 Terminal 11 Receiving unit 12 Generating unit 13 Transmitting unit 20 Management server 21 Receiving unit 22 Control execution unit 100 Remote operation system 110 Leader terminal 111 Physical environment 112 Virtual environment 113 HAV data processing unit 114 Encoder / decoder 115 QoE acquisition unit 120 Follower terminal 121 Physical environment 122 Virtual environment 123 HAV data processing unit 124 Encoder / decoder 125 QoE acquisition unit 130 Network 131 First network 132 Second network 140 SE server 141 First function unit 142 Second function unit 143 First model 144 Second model 145 Interface

Claims

1. A receiving unit that receives haptic data from other terminals connected via a network, A generation unit generates instruction information to a management server on the network, based on an indicator showing the communication status of the haptic data, to perform control over the communication of the haptic data with the other terminals. The system includes a transmission unit that transmits the instruction information to the management server. Terminal.

2. The generation unit predicts network parameters between the terminal and the other terminals based on the indicators and sends them to the management server, and generates instruction information to control communication between the terminal and the other terminals based on the predicted parameters. The terminal according to claim 1.

3. The generation unit predicts the haptic data generated on the other terminal side based on the indicator and generates instruction information to control the management server to change the content of the haptic data to be sent to the terminal and send it to the terminal based on the predicted haptic data. The terminal according to claim 1 or 2.

4. A receiving unit receives instruction information from a second terminal, which is connected to a first terminal via a network and receives haptic data from the first terminal, to perform control over the communication of the haptic data with the first terminal. A control execution unit that performs control related to the communication of the haptic data based on the instruction information, A management server equipped with the following features.

5. The control execution unit predicts network parameters between the first terminal and the second terminal based on the instruction information, and controls communication between the first terminal and the second terminal based on the predicted parameters. The management server according to claim 4.

6. The device and Equipped with a management server, The aforementioned terminal is A receiving unit that receives haptic data from other terminals connected via a network, A generation unit generates instruction information to the management server on the network, based on an indicator showing the communication status of the haptic data, for the management server to perform control over the communication of the haptic data with other terminals. It has a transmission unit that transmits the instruction information to the management server, The aforementioned management server A receiving unit that receives the instruction information from the terminal, The system includes a control execution unit that performs control related to the communication of the haptic data based on the instruction information. Communication system.

7. It receives haptic data from other devices connected via the network. Based on an indicator showing the communication status of the haptic data, instruction information is generated to the management server on the network to perform control over the communication of the haptic data with the other terminals. The instruction information is sent to the management server. How a computer can perform this task.

8. A second terminal, which is connected to a first terminal via a network and receives haptic data from the first terminal, receives instruction information from the second terminal to perform control over the communication of the haptic data with the first terminal. Based on the instruction information, control is performed regarding the communication of the haptic data. How a computer can perform this task.

9. It receives haptic data from other devices connected via the network. Based on an indicator showing the communication status of the haptic data, instruction information is generated to the management server on the network to perform control over the communication of the haptic data with the other terminals. The instruction information is sent to the management server. A program that causes a computer to perform a task.

10. A second terminal, which is connected to a first terminal via a network and receives haptic data from the first terminal, receives instruction information from the second terminal to perform control over the communication of the haptic data with the first terminal. Based on the instruction information, control is performed regarding the communication of the haptic data. A program that causes a computer to perform a task.