System, method, medium and equipment for realizing telemedicine video conference

Abstract

The invention discloses a system, a method, a medium and equipment for realizing a telemedicine video conference. The system comprises at least one first terminal equipment serving as near-end conference participant equipment of the video conference, at least one second terminal equipment serving as far-end conference participant equipment of the video conference, at least one controlled equipment and at least one first control equipment, the first terminal equipment comprises at least one controlled equipment interface and a plurality of network access interfaces, the first terminal equipment is connected with controlled equipment through the controlled equipment interface, and a multi-link transmission channel is formed through the plurality of network access interfaces of the first terminal equipment; the second terminal equipment comprises at least one remote control interface and at least one network access interface, the second terminal equipment is connected with the first control equipment through the remote control interface of the second terminal equipment, and the second terminal equipment is in network connection with the first terminal equipment through the network access interface of the second terminal equipment; according to the invention, the realization difficulty and cost of remote control of the remote medical video conference can be reduced, and the availability of the system can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to remote medical technology, and in particular, to a system for implementing a remote medical video conference, a method for implementing a remote medical video conference, a storage medium, and an electronic device. BACKGROUND

[0002] With the popularization of remote office, remote operation and maintenance, remote emergency command, and remote medical technology, video conferences have been more and more widely applied.

[0003] Some video conferences are relatively simple to implement, for example, each participant uses a mobile phone, a tablet computer, or a notebook computer or desktop computer to join the same video conference, and each participant can freely speak in the video conference or a participant with speaking right (such as a host) speaks in the video conference, thereby realizing the video conference.

[0004] However, some video conferences are relatively complex to implement, for example, in the application scenarios of remote medical treatment (such as remote surgery) and remote maintenance, various types of devices are often involved, such as multiple monitoring devices (such as vital sign monitoring devices) in an operating room and remotely controllable surgical robots and other controlled devices, which not only makes the connection between various types of devices complex, but also increases the complexity of implementing the video conference in the process of the video conference by the remote participant safely, conveniently, and accurately remotely controlling the surgical robots and other controlled devices.

[0005] How to conveniently hold a video conference in the application scenarios of remote medical treatment and remote maintenance and meet the needs of various types of devices in the video conference as much as possible to improve user experience is a technical problem worthy of attention. SUMMARY

[0006] To solve the above technical problem, the present disclosure is proposed. Embodiments of the present disclosure provide a system, a method, a storage medium, and an electronic device for implementing a remote medical video conference.

[0007] According to a first aspect of the embodiments of the present disclosure, a system for implementing a remote medical video conference is provided, the system comprising: at least one first terminal device as a near-end participant device of the video conference, at least one second terminal device as a far-end participant device of the video conference, at least one controlled device, and at least one first control device; the first terminal device comprising: at least one controlled device interface and a plurality of network access interfaces, the first terminal device being connected with the controlled device through the controlled device interface, and the first terminal device forming a multi-link transmission channel access network through the plurality of network access interfaces; the second terminal device comprising: at least one remote control interface and at least one network access interface, the second terminal device being connected with the first control device through the remote control interface, and the second terminal device forming a link transmission channel access network through the network access interface; the first terminal device being configured to collect controlled device multimedia information and near-end multimedia information, assign a first transmission priority to the controlled device multimedia information, assign a second transmission priority to the near-end multimedia information, assign a link transmission channel to the controlled device multimedia information and the near-end multimedia information according to the data transmission performance of each link transmission channel obtained by monitoring, the first transmission priority and the second transmission priority, and send the controlled device multimedia information and the near-end multimedia information to the second terminal device; wherein the first transmission priority is higher than the second transmission priority; the second terminal device being configured to receive first control information for the controlled device generated by the first control device through the remote control interface, collect far-end multimedia information, assign a third transmission priority to the first control information, assign a fourth transmission priority to the far-end multimedia information, and send the first control information and the far-end multimedia information to the first terminal device through the link transmission channel according to the third transmission priority and the fourth transmission priority; wherein the third transmission priority is higher than the fourth transmission priority; the first terminal device being further configured to receive the first control information and the far-end multimedia information from the second terminal device, send a control command to the controlled device through the controlled device interface according to the first control information, so that the controlled device performs a corresponding operation based on the control command, and play the far-end multimedia information in a video conference interface; and the second terminal device being further configured to receive the controlled device multimedia information and the near-end multimedia information from the first terminal device, and play the controlled device multimedia information and the near-end multimedia information in the video conference interface.

[0008] According to a second aspect of the embodiments of the present disclosure, a method for implementing a remote medical video conference is provided, the method comprising steps performed in a first terminal device as a near-end participant device of a video conference, and the first terminal device comprising: at least one controlled device interface, a plurality of network access interfaces, a first video conference module, a first multi-link aggregation transmission module, and a remote control module, the first terminal device being connected with the controlled device through the controlled device interface, and the first terminal device forming a multi-link transmission channel access network through the plurality of network access interfaces; the steps performed in the first terminal device as the near-end participant device of the video conference comprising: the first video conference module collecting controlled device multimedia information and near-end multimedia information, assigning a first transmission priority to the controlled device multimedia information, assigning a second transmission priority to the near-end multimedia information, and transmitting the controlled device multimedia information, the near-end multimedia information, the first transmission priority, and the second transmission priority to the first multi-link aggregation transmission module; wherein the first transmission priority is higher than the second transmission priority; the first multi-link aggregation transmission module monitoring data transmission performance of the multi-link transmission channel, and assigning a link transmission channel to the controlled device multimedia information and the near-end multimedia information according to the data transmission performance of the link transmission channel, the first transmission priority, and the second transmission priority, and transmitting the controlled device multimedia information and the near-end multimedia information to a second terminal device as a far-end participant device of the video conference; the first video conference module receiving first control information for the controlled device and far-end multimedia information from the second terminal device through the first multi-link aggregation transmission module, and transmitting the first control information to the remote control module, and playing the far-end multimedia information in a video conference interface; and the remote control module transmitting a control command to the controlled device through the controlled device interface according to the first control information from the first video conference module.

[0009] According to a third aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, the storage medium storing a computer program for implementing any of the above methods.

[0010] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing executable instructions of the processor; and the processor is configured to read the executable instructions from the memory and execute the instructions to implement any of the above methods.

[0011] The first video conference module in the first terminal device can not only collect and process the near-end multimedia information, but also collect and process the controlled device multimedia information, and the second video conference module in the second terminal device can not only collect and process the control information of the remote control device for the controlled device, but also collect and process the far-end multimedia information. In this way, in the case that one first terminal device is arranged at the near end and one second terminal device is arranged at the far end, the far end and the near end can efficiently form the video conference based on the remote control of the controlled device. The first multi-link aggregation transmission module in the first terminal device can monitor the multi-link transmission channel formed by the plurality of network access interfaces, and can allocate the link transmission channels with different data transmission performances to different information with different importance and transmit the information, so that the link transmission channel resources are combined with the quality service, and thus in the multi-task concurrent scene, the information related to the controlled device can be preferentially transmitted, which is beneficial to avoid the phenomenon that the remote end cannot timely understand the real-time situation of the controlled device due to the poor environment at the near end. In the case that the network line at the far end where the second terminal device is located has better data transmission performance, the second video conference module and the link transmission module in the second terminal device can also preferentially transmit the control information for the controlled device in the multi-task concurrent scene, which is beneficial to reduce the control delay of the control device for the controlled device. Therefore, the technical solution provided by the present disclosure combines the video conference technology, the remote control and the multi-link aggregation technology, which is beneficial to reduce the implementation difficulty, the implementation cost and the maintenance cost of the remote medical video conference involving a plurality of different types of devices, and is also beneficial to enable the control device at the far end to perform real-time and accurate effective control on the controlled device, thereby improving the practicability, the usability and the user experience of the remote control based on the video conference.

[0012] The technical solution of the present disclosure will be described in further detail below by means of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. The drawings provided in the present disclosure are used to provide further understanding of the embodiments of the present disclosure, and constitute a part of the specification, and are used to explain the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0014] Figure 1 An application scenario schematic diagram of the technical solution of the present disclosure for implementing the remote medical video conference;

[0015] Figure 2 a schematic diagram of an embodiment of a system for implementing a remote medical video conference according to the present disclosure;

[0016] Figure 3 a flow chart of an embodiment of a method for implementing a remote medical video conference according to the present disclosure;

[0017] Figure 4 a flow chart of another embodiment of a method for implementing a remote medical video conference according to the present disclosure;

[0018] Figure 5 a structural diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Example embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of embodiments of the present disclosure, and the present disclosure is not limited to the described example embodiments.

[0020] Summary of the disclosure

[0021] In the process of implementing the present disclosure, the inventors found that in many application scenarios, the environment in which a surgical robot, a medical examination device (such as an ultrasonic robot, a CT / DR robot, etc.), and an automated maintenance arm and other highly automated and remotely controllable devices (i.e., controlled devices) are located can be relatively undesirable, for example, an oxygen-deficient environment, a virus / air pollution environment, an abnormal temperature environment (such as a high-temperature / cold environment, etc.), a natural disaster environment, and a war environment, etc. Using a video conference to remotely control the controlled devices located in a poor environment at a remote end is beneficial to improving the convenience of operating the controlled devices. However, the joining of the controlled devices and the remote operation of the controlled devices increase the difficulty of organizing and holding a video conference, and often require some professional skills. Due to environmental restrictions and other factors, the environment in which the controlled devices are located often lacks professional skills, which on the one hand requires various controlled devices, auxiliary devices (such as vital sign monitoring devices and various sensors, etc.), and each terminal device as a participant to be able to simply and conveniently access the video conference, and on the other hand requires the video conference to ensure that the controlled devices can be effectively controlled in a timely and accurate manner.

[0022] Example overview

[0023] The technical solutions of the present disclosure for implementing a remote medical video conference can be applied to various application scenarios, such as a remote operation and maintenance and a remote medical application scenario that requires remote control of a controlled device. A specific example of an application scenario is shown in FIG. 1. Figure 1 ​

[0024] Figure 1 The operating room 100 in the embodiment can be a temporary operating room, such as a tent hospital, a warfront hospital, or a temporary hospital in an earthquake disaster. The operating room 100 is provided with a first terminal device 101, at least one controlled device (102-1 and 102-2 are shown in the embodiment), and at least one local device (103-1, 103-2, and 103-3 are shown in the embodiment). Figure 1 Figure 1 The operating room 100 in the embodiment can be a temporary operating room, such as a tent hospital, a warfront hospital, or a temporary hospital in an earthquake disaster. The operating room 100 is provided with a first terminal device 101, at least one controlled device (102-1 and 102-2 are shown in the embodiment), and at least one local device (103-1, 103-2, and 103-3 are shown in the embodiment).

[0025] The first terminal device 101 in the operating room 100 can be a participant device of a video conference, and the first terminal device 101 can be a host device of the video conference. In the case where the first terminal device 101 is provided with multiple controlled device interfaces, the first terminal device 101 can be connected to the controlled device 102-1 and the controlled device 102-2 at the same time, and in the case where the first terminal device 101 is provided with only one controlled device interface, the first terminal device 101 can be connected to only one of the controlled device 102-1 and the controlled device 102-2. In the case where the first terminal device 101 is provided with at least three audio and video collection interfaces, the first terminal device 101 can be connected to the local device 103-1, the local device 103-2, and the local device 103-3 at the same time by using the audio and video collection interfaces. In addition, the first terminal device 101 is also provided with multiple network access interfaces, and the first terminal device 101 can form a multi-link transmission channel by using the multiple network access interfaces, and can interact with other participants (such as experts 110 and 120) in the video conference through the multi-link transmission channel.

[0026] The controlled device 102-1 and the controlled device 102-2 in the operating room 100 can be surgical robots for performing two different types of surgeries (such as one being a brain surgery robot and the other being a heart surgery robot), or can be surgical robots for performing the same type of surgery (such as both being brain surgery robots or both being heart surgery robots). In one example, the two surgical robots can simultaneously perform two different types of surgeries on the same patient. In another example, the two surgical robots can also simultaneously perform the same type or two different types of surgeries on two patients.

[0027] The local device 103-1, the local device 103-2, and the local device 103-3 in the operating room 100 can be monitoring devices required for surgery, such as vital sign monitoring devices (such as electrocardiogram monitors), respiratory function monitoring devices (such as pulse oximetry monitors), and brain function monitoring devices (such as electroencephalogram monitors).​

[0028] Figure 1 The locations where the experts 110 and 120 are located are usually far away from the operating room 100, for example, the locations where the experts 110 and 120 are located can be expert offices of two first-class hospitals respectively. The location where the expert 110 is located is provided with a second terminal device 111 and a first control device 112, and the location where the expert 120 is located is provided with a second terminal device 121 and a first control device 122. The second terminal device 111 and the second terminal device 121 can both serve as participant devices of a video conference, and both the second terminal device 111 and the second terminal device 121 include at least one remote control interface and at least one network access interface.

[0029] The second terminal device 111 is connected with the first control device 112 through its remote control interface, and the second terminal device 111 accesses the network through its network access interface, so as to interact with other participants (such as the operating room 100 and the expert 120) in the video conference. In addition, in the case where the second terminal device 111 includes multiple network access interfaces, the second terminal device 111 can form a multi-link transmission channel by using the multiple network access interfaces, and interact with other participants in the video conference through the multi-link transmission channel. The expert 110 can control the controlled device (such as the controlled device 102-1 or the controlled device 102-2) in the operating room 100 to perform corresponding actions by operating the first control device 112. For example, assuming that the expert 110 needs to control the operation of the controlled device 102-1, when the expert 110 operates the first control device 112, the control information generated by the first control device 112 is preferentially transmitted to the first terminal device 101 in the operating room 100 through the link transmission channel with the best data transmission performance of the second terminal device 111, and the first terminal device 101 sends corresponding control commands (such as forward / backward / press down / lift up control commands of a mechanical hand) to the controlled device 102-1. The first terminal device 101 can preferentially transmit the real-time pictures from the controlled device 102-1 to the second terminal device 111 at the location of the expert 110 through the link transmission channel with the best data transmission performance, and the second terminal device 111 timely displays the pictures from the controlled device 102-1 in the video conference interface for the expert 110. The first terminal device 101 can also transmit the collected real-time pictures of the local device 103-1, the local device 103-2 and the local device 103-3 to the second terminal device 111 at the location of the expert 110 through the link transmission channel with the second best data transmission performance, and the second terminal device 111 timely displays the pictures from the local device 103-1, the local device 103-2 and the local device 103-3 in the video conference interface for the expert 110. In addition, the first terminal device 101 can also transmit the current environment pictures of the operating room 100 collected by it to the second terminal device 111 at the location of the expert 110 through the data transmission channel with poor transmission performance, and the second terminal device 111 timely displays the current environment pictures of the operating room 100 in the video conference interface for the expert 110.

[0030] The second terminal device 121 is connected with the first control device 122 through its remote control interface, and the second terminal device 121 accesses the network through its network access interface, so as to interact with other participants (such as the operating room 100 and the expert 110) in the video conference. In addition, in the case that the second terminal device 121 includes multiple network access interfaces, the second terminal device 121 can form a multi-link transmission channel by using the multiple network access interfaces, and interact with other participants in the video conference through the multi-link transmission channel. The expert 120 can control the controlled device (such as the controlled device 102-1 or the controlled device 102-2) in the operating room 100 to perform corresponding actions by operating the first control device 122. For example, assuming that the expert 120 needs to control the operation of the controlled device 102-2, when the expert 120 operates the first control device 122, the control information generated by the first control device 122 is preferentially transmitted to the first terminal device 101 in the operating room 100 through the link transmission channel with the best data transmission performance of the second terminal device 121, and the first terminal device 101 sends corresponding control commands (such as forward / backward / press down / lift up control commands of a mechanical arm) to the controlled device 102-2. The first terminal device 101 can preferentially transmit the real-time pictures from the controlled device 102-2 to the second terminal device 121 at the location of the expert 120 through the link transmission channel with the best data transmission performance, and the second terminal device 121 timely displays the pictures from the controlled device 102-2 in the video conference interface for the expert 120. The first terminal device 101 can also transmit the collected real-time pictures of the local device 103-1, the local device 103-2 and the local device 103-3 to the second terminal device 121 at the location of the expert 120 through the link transmission channel with the second best data transmission performance, and the second terminal device 121 timely displays the pictures from the local device 103-1, the local device 103-2 and the local device 103-3 in the video conference interface for the expert 120. In addition, the first terminal device 101 can also transmit the current environment pictures of the operating room 100 collected by it to the second terminal device 121 at the location of the expert 120 through the link transmission channel with the poorer data transmission performance, and the second terminal device 121 timely displays the current environment pictures of the operating room 100 in the video conference interface for the expert 120.

[0031] It should be particularly pointed out that the above Figure 1The following explanation uses the operating room 100 in the proximal / local end as an example. The operating room 100 in the proximal / local end can be replaced by an ultrasound workstation (the controlled device is an ultrasound robot), a CT / DR workstation (the controlled device is a CT / DR robot), or an X-ray workstation (the controlled device is a 3D fluoroscopic C-arm device), etc. In addition, in an application scenario, the proximal / local end can simultaneously include: operating room 100, ultrasound workstation, CT / DR workstation, and X-ray workstation, etc., and each workstation can be equipped with a first terminal device. The first terminal device is connected to the controlled device in the workstation through its controlled device interface, and the remote end of the system can simultaneously include: a first control device and a second terminal device set up for experts to remotely control the controlled devices in the workstation.

[0032] As can be seen from the above application scenario description, this disclosure sets up a first terminal device 101 at the near end and a second terminal device 111 at the far end, which can realize the remote control of the controlled device by the first control device using video conferencing. This not only greatly simplifies the implementation of video conferencing with multiple different types of devices, but also helps the first control device located at the far end to perform timely and accurate effective control of the controlled device.

[0033] Example system

[0034] Figure 2 This is a schematic diagram of one embodiment of a system for implementing telemedicine video conferencing according to the present disclosure. Figure 2 The system shown includes: a first terminal device 200 that can be used as a near-end participant device in a video conference, a second terminal device 210 that can be used as a remote participant device in a video conference, a controlled device 220, and a first control device 230. It should be noted that there can be multiple first terminal devices 200, second terminal devices 210, controlled devices 220, and first control devices 230. For clarity, this embodiment uses the system comprising a first terminal device 200, a second terminal device 210, a controlled device 220, and a first control device 230 as an example for illustration.

[0035] The first terminal device 200 is usually arranged at a near end, such as an operating room, a maintenance site, and the like. The first terminal device 200 includes at least one controlled device interface and a plurality of network access interfaces. The controlled device interface is used to connect the first terminal device 200 with the controlled device 220. For example, the first terminal device 200 is simultaneously connected with a plurality of controlled devices 220 located at a local place through a plurality of controlled device interfaces. The controlled device interface of the present disclosure is usually a wired controlled device interface, and of course, the present disclosure does not exclude the possibility that the controlled device interface is a wireless controlled device interface. For example, the controlled device interface can be specifically an RS232 (Recommended Standard) interface, an RS485 interface, a USB (Universal Serial Bus) interface, a Bluetooth interface, or an infrared interface, and the like, and the present disclosure does not limit the specific forms of the controlled device interface. The plurality of network access interfaces are used to connect the first terminal device 200 with a network such as a local area network or the Internet, that is, the first terminal device 200 can form a multi-link transmission channel through all or part of the network access interfaces, and perform information interaction with the second terminal device 210 through the multi-link transmission channel. All network access interfaces of the first terminal device 200 can include wired network access interfaces, such as fiber interfaces (such as SC interfaces or LC interfaces, and the like) and RJ45 (Registered Jack) interfaces, and the like, and can also include wireless network access interfaces, such as satellite communication-based interfaces, 4G / 5G cellular network-based interfaces, and WiFi-based interfaces, and the like.

[0036] The second terminal device 210 is usually located at a remote place, such as an expert office and a company headquarters. The second terminal device 210 comprises at least one remote control interface and at least one network access interface. The remote control interface is used to connect the second terminal device 210 with the first control device 230. For example, the second terminal device 210 can be connected with multiple first control devices 230 located at a local place through multiple remote control interfaces at the same time. The remote control interface of the present disclosure is usually a wired remote control interface, and can also be a wireless remote control interface. For example, the remote control interface can be an RS232 interface, an RS485 interface, a USB interface, a Bluetooth interface, or an infrared interface, and the specific form of the remote control interface is not limited in the present disclosure. The network access interface is used to access a network such as a local area network or the Internet. Since the environment where the second terminal device 210 is located usually has a good network environment, such as a dedicated optical fiber access, the second terminal device 210 can access the network through one network access interface. In the case that the second terminal device 210 accesses the network through multiple network access interfaces at the same time, the second terminal device 210 can form a multi-link transmission channel through all the network access interfaces, and interact with the first terminal device 200 through the multi-link transmission channel. The multiple network access interfaces of the second terminal device 210 can comprise wired network access interfaces, such as optical fiber interfaces (such as SC interfaces or LC interfaces, etc.) and RJ45 interfaces, and can also comprise wireless network access interfaces, such as satellite communication-based interfaces, 4G / 5G cellular network-based interfaces, and WiFi-based interfaces, etc.

[0037] The controlled device 220 is usually located at a near place together with the first terminal device 200. The controlled device 220 usually refers to a smart device (such as a surgical robot or a maintenance robot) that can perform corresponding operations according to received control commands. The controlled device 220 can not only perform corresponding operations according to the control commands transmitted by the first terminal device 200, but also can be connected with a local control device directly or through the first terminal device 200 in the case that a control device (such as the second control device described below) is located at a local place, and perform corresponding operations according to the control commands from the local control device.

[0038] The first control device 230 is usually arranged at the remote end together with the second terminal device 210, and the first control device 230 usually refers to a device for controlling the action of the controlled device 220. The first control device 230 can also be referred to as a simulation / simulation device, etc., that is, a device for simulating / simulating the action of the controlled device 220 to make the controlled device 220 perform the corresponding action. In an application scenario, the first control device 230 can simulate / simulate the action of the controlled device 220 by using a keyboard, a mouse, and / or a joystick, etc.

[0039] Since the controlled device 220 accesses the network through the first terminal device 200, the first control device 230 accesses the network through the second terminal device 210, and the control of the controlled device 220 by the first control device 230 is realized via the first terminal device 200 and the second terminal device 210, for the controlled device 220 and the first control device 230, the first terminal device 200 and the second terminal device 210 can be considered as the gateway of the controlled device 220 and the first control device 230, and thus in the medical field, the first terminal device 200 and the second terminal device 210 can be considered as the medical collaborative gateway. In addition, in the case that the first terminal device 200 and the second terminal device 210 each have multiple network access interfaces, the first terminal device 200 and the second terminal device 210 can be the same terminal device, that is, the terminal device at the near end is the first terminal device 200, and the terminal device at the remote end is the second terminal device 210, and the terminal device can become the device of the near-end participant in the video conference or the device of the remote-end participant according to its position, thereby performing the corresponding steps, and then becoming the first terminal device 200 or the second terminal device 210. Specifically, for a terminal device, if the terminal device is arranged at the near end and the terminal device is connected with the controlled device 220, the terminal device becomes the first terminal device 200 of the present disclosure, and if the terminal device is arranged at the remote end and the terminal device is connected with the first control device 230, the terminal device becomes the second terminal device 210 of the present disclosure.

[0040] The first terminal device 200 is mainly used for collecting multimedia information of the controlled device 220 connected thereto and near-end multimedia information, assigning a first transmission priority to the collected multimedia information of the controlled device and a second transmission priority to the collected near-end multimedia information, monitoring data transmission performance of each link transmission channel of the first terminal device 200, and assigning a link transmission channel to the multimedia information of the controlled device and the near-end multimedia information according to the data transmission performance of each link transmission channel obtained by monitoring, the first transmission priority and the second transmission priority, and sending the multimedia information of the controlled device and the near-end multimedia information to the second terminal device 210. The multimedia information of the controlled device refers to multimedia information formed by the controlled device, for example, video information of a patient's lesion or a position to be repaired provided by the controlled device. The multimedia information of the controlled device in the present disclosure can include video information with sound, etc. The near-end multimedia information refers to multimedia information of local devices and / or local environment other than the controlled device, and can include audio information and / or video information with sound, etc. The first transmission priority is higher than the second transmission priority, that is, the data transmission performance of the link transmission channel assigned by the first terminal device 200 to the multimedia information of the controlled device should not be lower than the data transmission performance of the link transmission channel assigned by the first terminal device 200 to the near-end multimedia information, so as to ensure that the multimedia information of the controlled device can be transmitted to the second terminal device 210 preferentially and with high quality.

[0041] The second terminal device 210 is mainly used for receiving the first control information for the controlled device 220 generated by the first manipulation device 230 through its remote control interface, and collecting the remote multimedia information. In the case that the second terminal device 210 accesses a network through a network access interface, the second terminal device 210 can directly assign a third transmission priority to the first control information and a fourth transmission priority to the remote multimedia information without monitoring the data transmission performance of the transmission channel formed by the network access interface, and the second terminal device 210 sends the first control information and the remote multimedia information to the first terminal device according to the third transmission priority and the fourth transmission priority. The first control information refers to the control information for controlling the action of the controlled device 220, for example, the first control information can be keyboard information such as left move key / right move key / up move key / down move key, or mouse left key / right key / mouse move information, or handle move information, etc. The remote multimedia information generally refers to the multimedia information of the remote environment, however, in the case that the video picture of some device in the remote needs to be provided to the near end, the remote multimedia information can include the collected multimedia information of the device in the remote. The remote multimedia information in the present disclosure can include audio information and / or video information with sound, etc. The third transmission priority should be higher than the fourth transmission priority, so as to ensure that the first control information can be transmitted to the first terminal device 200 in priority.

[0042] The first terminal device 200 is also mainly used for, in the case that the first control information and the remote multimedia information from the second terminal device 210 are received, not only sending the corresponding control command to the controlled device 220 through the controlled device interface according to the first control information, so that the controlled device 220 performs the corresponding operation based on the control command, but also playing the remote multimedia information in its video conference interface, so that the local staff can understand the remote situation, such as the staff present in the expert office, etc. In the case that the first control information is the control command that can be effectively recognized by the controlled device 220, the first terminal device 200 can directly send the control command to the controlled device 220, and in the case that the first control information is the control command that cannot be effectively recognized by the controlled device 220, the first terminal device 200 can first convert the first control information into the control command that can be recognized by the controlled device 220, and then send the control command to the controlled device 220 through the controlled device interface.

[0043] The second terminal device 210 is also mainly used for receiving the controlled device multimedia information and the near-end multimedia information from the first terminal device 200, and playing the controlled device multimedia information and the near-end multimedia information in the video conference interface of the second terminal device 210, so that the experts and other personnel at the far end can know the current situation of the controlled device and the object (such as the position of a lesion or the position to be repaired) and the near-end situation (such as the current environment of the near end and the current situation of other devices at the near end) in real time.

[0044] In one example, the first terminal device 200 can include at least one remote control interface which is the same as the remote control interface included in the second terminal device 210 (such as an RS232 interface, an RS485 interface, and a USB interface, and other wired-based remote control interfaces), so that the first terminal device 200 can be connected with the second control device 240 at the near end through the remote control interface, and thus the doctors, repairers and other related personnel at the near end can operate the second control device 240 to control the controlled device 220 through the first terminal device 200. The second control device 240 can also be called a simulation device, which simulates the action of the controlled device 220 to make the controlled device 220 perform the corresponding action. In one application scenario, the second control device 240 can simulate the action of the controlled device 220 by using a keyboard, a mouse and / or a joystick, and the first control device 230 and the second control device 240 can be control devices of the same type or control devices of different types which can both control the action of the controlled device 220.

[0045] In a specific example, in the case that the second control device 240 generates second control information for the controlled device 220 due to being manipulated, the first terminal device 200 receives the second control information transmitted by the second control device 240 through its remote control interface, and sends a control command to the controlled device 220 through the controlled device interface according to the second control information, so that the controlled device 220 performs corresponding operation based on the control command. In the case that the second control information is a control command that can be effectively recognized by the controlled device 220, the first terminal device 200 can directly send the control command to the controlled device 220, and in the case that the second control information is a control command that cannot be effectively recognized by the controlled device 220, the first terminal device 200 can first convert the second control information into a control command that can be recognized by the controlled device 220, and then send the control command to the controlled device 220 through the controlled device interface. Similarly, the second control information can be keyboard information such as left moving key / right moving key / up moving key / down moving key, can be mouse left key / right key / mouse moving information, can also be handle moving information, etc. In addition, it needs to be particularly pointed out that the second control device 240 can also be directly connected with the controlled device 220 without passing through the first terminal device 200, so that the second control information generated by the second control device 240 can be directly transmitted to the controlled device 220 through its controlled device interface. Whether the second control device 240 is connected with the controlled device 220 through the first terminal device 200 or is directly connected with the controlled device 220 through its controlled device interface, the controlled device multimedia information can be transmitted to the second terminal device 210 through the first terminal device 200, and played in the video conference interface by the second terminal device 210, so that the remote personnel can observe the manipulation of the controlled device 220 by the near-end personnel. By connecting the second control device 240 in the local with the remote control interface of the first terminal device 200, not only the controlled device 220 can be remotely manipulated by the first control device 230 in the remote, but also the controlled device 220 can be locally manipulated by the second control device 240 in the near-end, so that the switching between local manipulation and remote manipulation is possible, the remote personnel not only can observe the manipulation of the controlled device 220 by the near-end personnel, but also the experts and other personnel in the remote can provide corresponding technical guidance to the near-end personnel through the video conference, thereby being beneficial to improve the control flexibility of the controlled device 220.

[0046] In one example, when the second terminal device 210 includes multiple network access interfaces and forms a multi-link transmission channel through these interfaces (i.e., the second terminal device 210 interacts with the first terminal device 200 through the multi-link transmission channel), the second terminal device 210 should monitor the data transmission performance of each link transmission channel. Based on the monitored data transmission performance, third transmission priority, and fourth transmission priority of each link transmission channel, the second terminal device 210 should allocate link transmission channels for the first control information and the remote multimedia information, respectively. Then, using the allocated link transmission channels, the second terminal device 210 should send the first control information and the remote multimedia information to the first terminal device 200. The data transmission performance of the link transmission channel allocated by the second terminal device 210 for the first control information should be no less than that of the link transmission channel allocated for the remote multimedia information, to ensure that the first control information can be transmitted to the first terminal device 200 with priority and reliability, thereby improving the real-time performance and reliability of remote control of the controlled device 220.

[0047] The first terminal device 200 and the second terminal device 210 in this disclosure can both include: a video conferencing module, a multi-link aggregation transmission module, and a remote control module. The video conferencing module can include: a first video conferencing module and a second video conferencing module. The multi-link aggregation transmission module can include: a first multi-link aggregation transmission module and a second multi-link aggregation transmission module. The operations performed by each module can be found in the relevant descriptions in the following method embodiments, and will not be detailed here.

[0048] Example method

[0049] Figure 3 This is a flowchart illustrating one embodiment of the method for implementing telemedicine video conferencing disclosed herein. Figure 3 The method shown includes steps S300, S301, S302, and S303. Figure 3 Each step in the process is executed on a terminal device, which is the near-end participant device in the video conference, i.e. Figure 3 Each step in the process is executed in a first terminal device, which serves as the near-end participant's device in the video conference. This first terminal device includes not only at least one controlled device interface and multiple network access interfaces, but also a first video conferencing module, a first multi-link aggregation transmission module, and a remote control module to execute the corresponding steps. Near-end users can initiate video conferences using the first terminal device, and can also join existing video conferences using the first terminal device. The processes for initiating and joining video conferences are the same as existing processes and will not be described in detail here. The following sections... Figure 3 Each step in the process will be explained separately.

[0050] S300, the first video conference module collects the multimedia information of the controlled device and the near-end multimedia information, assigns a first transmission priority to the multimedia information of the controlled device and a second transmission priority to the near-end multimedia information, and transmits the multimedia information of the controlled device, the near-end multimedia information, the first transmission priority, and the second transmission priority to the first multi-link aggregation transmission module.

[0051] The first video conference module can identify the collected information to determine whether the collected information is the multimedia information of the controlled device or the near-end multimedia information. For example, the first video conference module can identify the collected information through the channel of the collected information, and the specific implementation of the first video conference module to identify the collected information is not limited in the present disclosure. The first transmission priority and the second transmission priority assigned by the first video conference module to the multimedia information of the controlled device and the near-end multimedia information can be corresponding flag bits. For example, the first video conference module inserts corresponding flag bits in the multimedia information of the controlled device and the near-end multimedia information.

[0052] The near-end multimedia information collected by the first video conference module of the present disclosure can include at least one of the local device multimedia information and the current environment multimedia information. In the case where the near-end multimedia information includes the local device multimedia information and the current environment multimedia information, the second transmission priority of the present disclosure can include two different transmission priorities, both of which are lower than the first transmission priority assigned to the multimedia information of the controlled device. That is, the transmission priorities assigned by the first video conference module to the local device multimedia information and the current environment multimedia information can be different, and the transmission priority assigned to the local device multimedia information is generally higher than the transmission priority assigned to the current environment multimedia information.

[0053] In one example, the first video conference module can have an encoding processing capability, i.e., the first video conference module respectively encodes the collected multimedia information of the controlled device and the near-end multimedia information, and provides the encoded multimedia information of the controlled device and the near-end multimedia information and the corresponding transmission priorities to the first multi-link aggregation transmission module. In addition, the first video conference module can also have a preprocessing capability, i.e., before encoding processing, the first video conference module can first preprocess the collected multimedia information of the controlled device and the near-end multimedia information, and then the first video conference module respectively encodes the preprocessed multimedia information of the controlled device and the near-end multimedia information, and provides the encoded multimedia information of the controlled device and the near-end multimedia information and the corresponding transmission priorities to the first multi-link aggregation transmission module. The preprocessing can include at least one of noise reduction processing and echo cancellation processing.

[0054] In one example, in the case that the first terminal device comprises a pre-processing unit (such as a pre-processing chip) and a codec unit (such as a codec), the pre-processing operation and the encoding processing operation can be performed by the pre-processing unit and the codec unit, and the first video conference module provides the transmission priority, the controlled device multimedia information and the near-end multimedia information to the first multi-link aggregation transmission module together after assigning transmission priorities to the controlled device multimedia information and the near-end multimedia information output by the codec processing unit.

[0055] S301, the first multi-link aggregation transmission module monitors the data transmission performance of the multi-link transmission channels formed by the multiple network access interfaces, and assigns link transmission channels to the controlled device multimedia information and the near-end multimedia information according to the data transmission performance of each link transmission channel, the first transmission priority and the second transmission priority, and sends the controlled device multimedia information and the near-end multimedia information to the second terminal device as a remote participant device of the video conference.

[0056] The data transmission performance of the link transmission channel assigned by the first multi-link aggregation transmission module in the disclosure to the controlled device multimedia information with the first transmission priority is generally better than the data transmission performance of the link transmission channel assigned to the near-end multimedia information with the second transmission priority. The data transmission performance of the link transmission channel in the disclosure can be reflected by indicators such as transmission delay (Delay), packet loss rate (Packet Loss) and jitter (Jitter). In addition, the first multi-link aggregation transmission module can determine the type of data suitable for transmission of each link transmission channel according to the monitored indicators of each link transmission channel, so as to perform the assignment of the link transmission channel.

[0057] In one application scenario, the link transmission channel assigned by the first multi-link aggregation transmission module to the controlled device multimedia information generally has the characteristics of low transmission delay, small packet loss rate and small jitter, while the link transmission channel assigned by the first multi-link aggregation transmission module to the near-end multimedia information can be slightly worse in terms of transmission delay, packet loss rate and jitter. The first multi-link aggregation transmission module can determine the data transmission performance of each link transmission channel by using the existing monitoring method, which will not be described in detail here.

[0058] Since the controlled device multimedia information and the near-end multimedia information can be transmitted through different link transmission channels, in the multi-task concurrent scenario, it is beneficial to avoid the occupation of the link transmission resource by the near-end multimedia information and the like to affect the timely and reliable transmission of the controlled device multimedia information, and since the controlled device multimedia information is transmitted through a link transmission channel with better data transmission performance, it is further beneficial to ensure the timely and reliable transmission of the controlled device multimedia information.

[0059] S302, the first video conference module receives the first control information for the controlled device and the remote multimedia information from the second terminal device through the first multi-link aggregation transmission module, transmits the first control information to the remote control module, and plays the remote multimedia information in the video conference interface.

[0060] In one example, when the first multi-link aggregation transmission module receives information through the multi-link transmission channel, the received information can be provided to the first video conference module, and the first video conference module decodes and processes the received information, and identifies the decoded and processed information. For the identified first control information for the controlled device from the second terminal device, the first video conference module transmits the first control information to the remote control module. For the identified remote multimedia information from the second terminal device, the first video conference module performs the operation of playing the remote multimedia information in the video conference interface, for example, playing the remote multimedia information in a specific area in the video conference interface.

[0061] In another example, when the first multi-link aggregation transmission module receives information through the multi-link transmission channel, the received information is provided to a codec unit (such as a codec) in the first terminal device, and the codec unit decodes and processes the received information. The first video conference module receives the decoded and processed information output by the codec unit, and identifies the decoded and processed information. For the identified first control information for the controlled device from the second terminal device, the first video conference module transmits the first control information to the remote control module. For the identified remote multimedia information from the second terminal device, the first video conference module performs the operation of playing the remote multimedia information in the video conference interface, for example, playing the remote multimedia information in a specific area in the video conference interface.

[0062] In one application scenario, the first video conference module can prioritize the identified first control information for the controlled device from the second terminal device for processing, to ensure that the first control information can be transmitted to the remote control module as soon as possible.

[0063] In one example, the first control information in the present disclosure can be control information based on JSON (JavaScript Object Notation) or XML (Extensible Markup Language), and the first control information can include control commands and their parameters, etc.

[0064] S303, the remote control module sends a control command to the controlled device through the controlled device interface according to the first control information from the first video conference module.

[0065] When the first control information is a control command that can be effectively recognized by the controlled device, the remote control module can directly transmit the first control information to the controlled device through the controlled device interface. When the first control information is not a control command that can be effectively recognized by the controlled device, the remote control module can first convert the first control information into a control command that can be effectively recognized by the controlled device, and then transmit the converted control command to the controlled device through the controlled device interface. In addition, the operation of judging whether the first control information is a control command that can be effectively recognized by the controlled device and converting the first control information into a control command that can be effectively recognized by the controlled device can also be performed by the first video conference module.

[0066] In one application scenario, the first video conference module of the present disclosure can also be used to control the operation permission of the controlled device from the far end to the near end. Specifically, the first video conference module can set the controlled device sharing control permission information according to the user's operation, where the user is usually a user with management permission, for example, in the case where the user logs in as an administrator, the first video conference module provides a control permission setting interface for the user (such as setting a "shared device" button on the video conference interface, when the button is clicked, the first video conference module displays the control permission setting interface for the user), the user sets the operation permission of the controlled device for each remote user through the operation of the control permission setting interface, of course, the operation permission of the controlled device for the near-end user can also be set, and the first video conference module generates the controlled device sharing control permission information. Different remote users can have different operation permissions, for example, some remote users have partial control permissions of the controlled device, and some remote users have all control permissions of the controlled device, thereby facilitating the refinement of the operation permission management of the remote users. In one example, all the controlled device sharing control permission information set by the first video conference module can correspond to at least one level of shared control permission, and the control commands corresponding to different levels of shared control permission information are usually different, for example, the highest level of shared control permission information corresponds to all control commands of the controlled device, and the shared control permission information of other levels corresponds to part of the control commands of the controlled device. The controlled device sharing control permission information in the present disclosure can include the correspondence between the remote user identifier (i.e. the remote participant user identifier), the shared control permission level, and the allowed control command set. The user with management permission in the present disclosure can adjust the controlled device sharing control permission information at any time, for example, temporarily setting the corresponding shared control permission information for a certain remote user, canceling the shared control permission of a certain remote user, and modifying the shared control permission information of a certain remote user after the video conference has been held for a period of time, thereby facilitating more flexible permission management.

[0067] In the case that the controlled device sharing control permission information is set, when the first video conference module receives a sharing control request from the second terminal device (e.g., a remote user clicks a remote control button in the video conference interface), the first video conference module can use the set controlled device sharing control permission information to verify the sharing control permission of the sharing control request. The first video conference module should return the sharing control permission verification result to the second terminal device. For example, the first video conference module can obtain the remote user identifier and the requested sharing control permission level from the sharing control request, and check whether the remote user identifier and the requested sharing control permission level belong to the controlled device sharing control permission information. If they belong to the controlled device sharing control permission information, the verification result is verified successfully. If they do not belong to the controlled device sharing control permission information, the verification result is verified unsuccessfully. The first video conference module can assign a third transmission priority to the verification result, and transmit the verification result and the third transmission priority to the first multi-link aggregation transmission module. The first multi-link aggregation transmission module assigns a link transmission channel to the verification result from the first video conference module according to the data transmission performance of each link transmission channel obtained by monitoring and the received third transmission priority, and sends the verification result to the second terminal device through the assigned link transmission channel. The third transmission priority in the present disclosure is usually not lower than the first transmission priority, i.e., the third transmission priority can be better than or equal to the first transmission priority. By setting a remote control button in the video conference interface to trigger the sending of the sharing control request, each participant in the video conference can select and control the controlled device in the video conference at any time according to the actual situation, thereby facilitating the flexible implementation of the sharing control of the controlled device.

[0068] In one example, when the first video conference module receives the first control information from the second terminal device, it should determine whether the first control information is within the shared control authority range of the second terminal device, for example, the first video conference module first determines the shared control authority level of the sender of the first control information by using the above-mentioned correspondence relationship and the sender identifier of the first control information, and then determines whether the first control information belongs to the set of allowed control commands corresponding to the shared control authority level of the sender. If it belongs to the set of control commands, the first video conference module transmits the first control information to the remote control module, and the remote control module sends the corresponding control command to the controlled device through the controlled device interface according to the first control information. If it does not belong to the set of control commands, the first video conference module can discard the first control information. In addition, when the first video conference module discards the first control information, it can generate corresponding response information, which should be returned to the second terminal device through the first multi-link aggregation transmission module. Furthermore, after the first video conference module sends the first control information to the remote control module, it will receive the response information returned by the remote control module to it whether the control command is successfully issued to the controlled device. The first video conference module can assign a transmission priority to the response information it generates / receives, and provide the response information and the transmission priority to the first multi-link aggregation transmission module. The first multi-link aggregation transmission module assigns a link transmission channel to the response information according to the transmission priority, and returns the response information to the second terminal device through the assigned link transmission channel. The transmission priority assigned to the response information can not be lower than the transmission priority of the controlled device multimedia information, for example, higher than or the same as the transmission priority of the controlled device multimedia information.

[0069] In another example, when the first video conference module receives the first control information from the second terminal device, the first control information can be directly provided to the remote control module, and the remote control module determines whether the first control information is within the shared control authority range of the second terminal device. For example, the remote control module first determines the shared control authority level of the sender of the first control information by using the above-mentioned correspondence relationship and the sender identifier of the first control information, and then determines whether the first control information belongs to the set of control commands allowed to be used corresponding to the shared control authority level of the sender. If the first control information belongs to the set of control commands allowed to be used, the remote control module sends a control command to the controlled device through the controlled device interface according to the first control information. If the first control information does not belong to the set of control commands allowed to be used or it is determined that the sender does not have the shared control authority level, the remote control module can discard the first control information. In addition, whether the remote control module sends a control command to the controlled device or discards the first control information, the remote control module can return corresponding response information to the first video conference module. The first video conference module can assign a transmission priority to the response information from the remote control module and provide the response information and the transmission priority to the first multi-link aggregation transmission module. The first multi-link aggregation transmission module assigns a link transmission channel to the response information according to the transmission priority, and returns the response information to the second terminal device through the assigned link transmission channel. The transmission priority assigned to the response information can be no lower than the transmission priority of the controlled device multimedia information, for example, higher than or the same as the transmission priority of the controlled device multimedia information.

[0070] By setting the controlled device shared control authority information for the remote user, the remote control of the controlled device is limited to the remote user with corresponding authority, which is beneficial to avoid the phenomenon that the controlled device is illegally remotely controlled by an unauthorized user, thereby being beneficial to ensure the safety of the remote control of the controlled device.

[0071] In one example, the proximal multimedia information of the present disclosure comprises at least one of local device multimedia information and current environment multimedia information. The local device can comprise a device disposed at the proximal end and facilitating assisting the remote user in remotely operating the controlled device, such as a vital sign monitoring device, etc. The current environment refers to an environment where the controlled device is located, such as an operating room or a machine room where the device to be maintained is located, etc. The first terminal device can comprise at least one of an audio / video acquisition interface and a microphone camera unit, and the first terminal device is connected with the local device through the audio / video acquisition interface, so that the first video conference module can acquire the local device multimedia information, and the first video conference module can acquire the current environment multimedia information through the microphone camera unit. The audio / video acquisition interface can be a HDMI (High Definition Multimedia Interface) or the like.

[0072] In one example, the first video conference module pre-processes (such as noise reduction processing and echo cancellation processing, etc.) the local device multimedia information acquired through the audio / video acquisition interface, encodes the pre-processed local device multimedia information, and then assigns a transmission priority to the local device multimedia information and provides the local device multimedia information to the first multi-link aggregation transmission module. The first video conference module can also pre-process (such as noise reduction processing and echo cancellation processing, etc.) the current environment multimedia information acquired through the microphone camera unit, encode the pre-processed current environment multimedia information, and then assign a transmission priority to the current environment multimedia information and provide the current environment multimedia information to the first multi-link aggregation transmission module.

[0073] In another example, the pre-processing unit in the first terminal device pre-processes (such as noise reduction processing and echo cancellation processing, etc.) the local device multimedia information collected through the audio-video collection interface, and then the codec in the first terminal device encodes the pre-processed local device multimedia information output by the pre-processing unit, the first video conference module receives the encoded local device multimedia information output by the codec, and assigns a transmission priority to the local device multimedia information and provides the local device multimedia information and the transmission priority to the first multi-link aggregation transmission module. Similarly, the pre-processing unit in the first terminal device pre-processes (such as noise reduction processing and echo cancellation processing, etc.) the current environment multimedia information transmitted by the microphone camera unit, and then the codec in the first terminal device encodes the pre-processed current environment multimedia information output by the pre-processing unit, the first video conference module receives the encoded current environment multimedia information output by the codec, and assigns a transmission priority to the current environment multimedia information and provides the current environment multimedia information and the transmission priority to the first multi-link aggregation transmission module.

[0074] In the above two examples, the transmission priority assigned by the first video conference module to the local device multimedia information is generally higher than the transmission priority assigned by the first video conference module to the current environment multimedia information, so that the local device multimedia information can be transmitted to the second terminal device in priority and reliably.

[0075] By obtaining the local device multimedia information through the audio-video collection interface and obtaining the current environment multimedia information through the microphone camera unit, the remote user can have a full understanding of the near-end situation, which is conducive to assisting the remote user in remotely and accurately controlling the near-end controlled device. By setting transmission priorities for the local device multimedia information and the current environment multimedia information, respectively, the local device multimedia information can be transmitted to the second terminal device in priority over the current environment multimedia information, so as to facilitate ensuring the transmission quality of the local device multimedia information which is more important to the remote user, and thereby facilitating better assisting the remote user in remotely controlling the controlled device.

[0076] In one example, the first terminal device of the present disclosure is further provided with at least one remote control interface, and the first terminal device is connected with the second control device through the remote control interface. The second control device is generally a device for controlling the action of the controlled device. The second control device can also be referred to as a simulation device, that is, the second control device can be used to realize remote simulation control of the controlled device. The first control device and the second control device can be control devices of the same model, that is, the control device located at the near end is the second control device, and the control device located at the far end is the first control device. The first video conference module can receive the second control information generated by the second control device for the controlled device through the remote control interface of the first terminal device, and transmit the second control information to the remote control module by the first video conference module, so that the remote control module can send control commands to the controlled device through the controlled device interface according to the second control information from the first video conference module. In addition, the first video conference can assign a transmission priority to the second control information, and provide the second control information and the transmission priority to the first multi-link aggregation transmission module, and the first multi-link aggregation transmission module assigns a link transmission channel to the second control information based on the data transmission performance of each link transmission channel obtained by monitoring and the transmission priority, and sends the second control information to the second terminal device at the far end using the assigned link transmission channel. The second video conference module in the second terminal device can display the corresponding picture in the video conference interface based on the second control information, so that the experts and other users at the far end can fully understand the operation of the second control device at the near end for the controlled device. In the above description, the encoding process of the second control information is omitted. The encoding process of the second control information can be performed by the first video conference module or the codec unit in the first terminal device.

[0077] By setting the second control device and processing the second control information generated by the second control device by the first video conference module, the first video conference module can know all the actual operations of the second control device, so that the first video conference module can display all the actual operations of the second control device in the video conference interface of the first terminal device, and the second video conference module in the second terminal device can also know all the actual operations of the second control device and display all the actual operations of the second control device in the video conference interface of the second terminal device, thereby facilitating the users at the far end to fully understand the operation of the users at the near end and give timely guidance to the operation of the users at the near end.

[0078] In one example, if the terminal device is used as a far-end participant device of a video conference (i.e., the terminal device is arranged at a far end such as an expert room), the terminal device is the second terminal device in the present disclosure, and if the terminal device is used as a near-end participant device of a video conference (i.e., the terminal device is arranged at a near end such as an operating room), the terminal device is the first terminal device in the present disclosure. Therefore, the second terminal device also includes at least one far-end control interface and at least one network access interface, the second terminal device is connected with the first control device through the far-end control interface, in the case that the second terminal device can access the network through a high-quality data transmission channel such as a fiber optic line, the second terminal device can access the network through only one network access interface, and in the case that the quality of the network accessed by the second terminal device is substandard, the second terminal device can access the network through multiple network access interfaces at the same time, which is beneficial to effectively ensure the timeliness and reliability of information transmission in a multi-task concurrent scenario.

[0079] For the second terminal device, the method for implementing video transmission in the present disclosure has a flow as shown in Figure 4 The method shown in Figure 4 includes steps S400, S401, S402 and S403, i.e. Figure 4 The steps contained are executed by modules in the second terminal device. The far-end user can initiate a video conference by using the second terminal device, or can join an existing video conference by using the second terminal device. The process of initiating a video conference and joining an existing video conference through the second terminal device can adopt an existing process, which will not be described in detail here. The steps in the figure will be described below respectively.

[0080] S400, the second video conference module receives, through its remote control interface, first control information generated by the first control device for the controlled device, and can collect far-end multimedia information.

[0081] The first control information received by the second video conference module can include keyboard-based key information, mouse-based click information and joystick-based movement information, etc. The far-end multimedia information collected by the second video conference module can be far-end current environment multimedia information collected through a microphone camera unit in the second terminal device, thereby facilitating the near-end user to fully understand the far-end situation.

[0082] In one example, the second video conference module can pre-process (such as noise reduction processing and echo cancellation processing, etc.) the far-end current environment multimedia information collected through the microphone camera unit, and encode the pre-processed far-end current environment multimedia information.

[0083] S401, the second video conference module assigns a third transmission priority to the first control information, assigns a fourth transmission priority to the remote multimedia information, and transmits the first control information, the remote multimedia information, the third transmission priority, and the fourth transmission priority to the link transmission module.

[0084] The third transmission priority is higher than the fourth transmission priority in the present disclosure, and the transmission priority assigned by the second video conference module to the first control information can be the highest priority of the local end. The second video conference module can preferentially process the first control information, for example, preferentially providing the first control information with the third transmission priority to the link transmission module.

[0085] In one example, the second video conference module can perform encoding processing on the first control information and the remote multimedia information respectively, and provide the first control information and the remote multimedia information after encoding processing to the link transmission module.

[0086] In another example, the operation of performing encoding processing on the first control information and the remote multimedia information can be performed by a codec unit in the second terminal device, for example, the codec unit performs encoding processing on the first control information and the remote multimedia information output by the second video conference module, and the first control information and the remote multimedia information after encoding processing output by the codec unit are provided to the link transmission module.

[0087] S402, the link transmission module sends the first control information and the remote multimedia information to the first terminal device through the network access interface according to the third transmission priority and the fourth transmission priority.

[0088] In one example, in the case where the second terminal device is provided with multiple network access interfaces, and the second terminal device only utilizes one network access interface (such as a dedicated optical fiber interface, etc.) to form a single link transmission channel, the link transmission module can preferentially process the first control information, for example, preferentially sending the first control information to the first terminal device to ensure that the first control information can be timely transmitted to the first terminal device. In addition, the link transmission module can monitor the data transmission performance of the link transmission channel, and when the data transmission performance of the link transmission channel is poor (such as not meeting the preset transmission performance requirement), information about the poor data transmission performance can be transmitted to the second video conference module, and the second video conference module prompts the user that the second terminal device needs to access the network through multiple network access interfaces.

[0089] In one example, in the case that the second terminal device is provided with multiple network access interfaces, and the multiple network access interfaces form multiple link transmission channels for the second terminal device, the second terminal device simultaneously accesses the network through the multiple network access interfaces, at this time, the link transmission module in the second terminal device can be considered as a second multi-link aggregation transmission module. The second multi-link aggregation transmission module is specifically configured to monitor the data transmission performance of each link transmission channel, and the data transmission performance of each link transmission channel can be reflected by performance indexes such as transmission delay, packet loss rate and jitter. The data transmission performance of the link transmission channel allocated by the second multi-link aggregation transmission module for the first control information with the third transmission priority is generally better than the data transmission performance of the link transmission channel allocated by the second multi-link aggregation transmission module for the remote multimedia information with the fourth transmission priority. That is, the link transmission channel allocated by the second multi-link aggregation transmission module for the first control information generally has characteristics such as low transmission delay, small packet loss rate and small jitter, while the link transmission channel allocated by the second multi-link aggregation transmission module for the remote multimedia information can be slightly worse in terms of transmission delay, packet loss rate and jitter. In addition, the link transmission channel allocated by the second multi-link aggregation transmission module for the first control information can be the link transmission channel with the best data transmission performance at the local end. The second multi-link aggregation transmission module can determine the data transmission performance of each link transmission channel by using an existing monitoring method, which will not be described in detail here.

[0090] Since the first control information can be given priority transmission processing, in the multi-task concurrent scenario, it is beneficial to avoid the occupation of the link transmission resource by the remote multimedia information and the like affecting the timely and reliable transmission of the first control information. Further, in the case that the first control information and the remote multimedia information can be transmitted through different link transmission channels, the first control information can be transmitted through the link transmission channel with the best data transmission performance, which is further beneficial to ensure the timely and reliable transmission of the first control information.

[0091] In one example, the second multi-link aggregation transmission module can allocate link transmission channels for the first control information and the remote multimedia information based on the load balancing principle, so as to avoid the phenomenon that the link transmission channel with good data transmission performance becomes poor due to being allocated with too much data to be transmitted, thereby affecting the timely and reliable transmission of the first control information. In addition, the multi-link aggregation transmission module of the present disclosure can allocate multiple link transmission channels (such as usually two link transmission channels) for the first control information based on the load balancing principle, one of which (such as the link transmission channel with the best data transmission performance) can be considered as a basic link transmission channel, and the others (such as the link transmission channels with the next best data transmission performance) can be considered as redundant link transmission channels. By allocating the basic link transmission channel and the redundant link transmission channel for the first control information, it is beneficial to avoid the phenomenon that the first control information transmission is abnormal (such as being blocked) due to factors such as sudden abnormality (such as channel congestion / packet loss / delay) of the link transmission channel, thereby facilitating the timely and reliable transmission of the first control information.

[0092] In one example, the present disclosure can pre-set a transmission performance requirement for the first control information, which can be considered as a link transmission channel switching trigger condition. In the case that the data transmission performance of the link transmission channel currently used to transmit the first control information does not meet the transmission performance requirement, the second multi-link aggregation transmission module can perform a link transmission channel switching operation for the first control information according to the link transmission channel currently monitored to meet the transmission performance requirement. In addition, the load balancing principle should also be considered in the process of performing the link transmission channel switching operation, for example, in the case that the data transmission performances of the two link transmission channels are the same, the link transmission channel with lighter load can be used as the target link transmission channel for switching.

[0093] By setting the transmission performance requirement for the first control information, it is beneficial to switch the link transmission channel for the first control information when the data transmission performance of the current link transmission channel used to transmit the first control information does not meet the requirement, and it is beneficial to make the link transmission channel for the first control information always have good data transmission performance. By considering the load balancing principle when performing the link transmission channel switching, it is beneficial to avoid the phenomenon that the switched link transmission channel becomes overloaded again due to the increase of data transmission, thereby causing the switching of the link transmission channel again.

[0094] S403, the second video conference module receives the controlled device multimedia information and the near-end multimedia information from the first terminal device through the network access interface thereof, and plays the controlled device multimedia information and the near-end multimedia information in the video conference interface thereof.

[0095] In one example, when the second multi-link aggregation transmission module receives information through the multi-link transmission channel, the second multi-link aggregation transmission module provides the received information to the second video conference module. The second video conference module decodes the received information and identifies the decoded information. For the identified controlled device multimedia information from the first terminal device, the current environment multimedia information of the near end, and the local device multimedia information of the near end, the second video conference module performs an operation of playing corresponding multimedia information in the video conference interface, for example, the second video conference module plays the controlled device multimedia information, the current environment multimedia information of the near end, and the local device multimedia information of the near end in a plurality of specific regions in the video conference interface, respectively.

[0096] In another example, when the second multi-link aggregation transmission module receives information through the multi-link transmission channel, the second multi-link aggregation transmission module can provide the received information to a codec unit in the second terminal device. The codec unit decodes the received information. The second video conference module identifies the decoded information output by the codec unit. For the identified controlled device multimedia information from the first terminal device, the current environment multimedia information of the near end, and the local device multimedia information of the near end, the second video conference module performs an operation of playing corresponding multimedia information in the video conference interface, for example, the second video conference module plays the controlled device multimedia information, the current environment multimedia information of the near end, and the local device multimedia information of the near end in a plurality of specific regions in the video conference interface, respectively.

[0097] In one application scenario, the second video conference module can prioritize the identified controlled device multimedia information from the first terminal device to ensure that the controlled device multimedia information can be played in the video conference interface in a timely manner.

[0098] In one example, after the remote user completes the remote operation on the controlled device, the remote user can click an end control button in the video conference interface. The second video conference module can send end remote control information to the first terminal device through the second multi-link aggregation transmission module, so that the first video conference module in the first terminal device can display corresponding information in the video conference interface, so that the near-end user knows that the remote operation of the remote user is completed. In addition, the first video conference module can release the control authority of the remote user on the controlled device to ensure the security of the control authority. The second video conference module can set a transmission priority (such as a third transmission priority) for the end remote control information, and the second multi-link aggregation transmission module can allocate a corresponding link transmission channel for the end remote control information based on the transmission priority.

[0099] Example electronic device

[0100] The following will be described with reference toFigure 5 An electronic device according to embodiments of the present disclosure is described. Figure 5 A block diagram of an electronic device according to embodiments of the present disclosure is shown. As shown, the electronic device 51 includes one or more processors 511 and a memory 512. Figure 5 The processor 511 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 51 to perform desired functions.

[0101] The processor 511 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 51 to perform desired functions.

[0102] The memory 512 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk drives, solid-state drives, and / or the like. The computer-readable storage media can store one or more computer program instructions, which can be executed by the processor 511 to implement the methods described above for implementing a telemedicine video conference and / or other desired functions of the various embodiments of the present disclosure.

[0103] In one example, the electronic device 51 can further include an input device 513, an output device 514, and / or the like, which can be interconnected through a bus system and / or other form of connection mechanism (not shown). The input device 513, for example, can include a keyboard, a mouse, and / or the like. The output device 514 can output various information externally. The output device 514 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0104] Of course, for simplicity, Figure 5 In the description of the electronic device 51, only some of the components related to the present disclosure are shown, and components such as buses, input / output interfaces, and the like are omitted. In addition, the electronic device 51 can include any other appropriate components, depending on the specific application.

[0105] Example computer program product and computer readable storage medium

[0106] In addition to the above-described methods and devices, embodiments of the present disclosure can also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods for implementing a telemedicine video conference according to various embodiments of the present disclosure described in the above "Exemplary Methods" section of the specification.

[0107] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0108] Furthermore, embodiments of the present disclosure can also be a computer readable storage medium, having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the steps described in the above "Exemplary Method" section of the present specification for implementing the method for implementing remote medical video conference according to various embodiments of the present disclosure.

[0109] The computer readable storage medium can be any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium can include the following: an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0110] The above describes the basic principles of the present disclosure in combination with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present disclosure are only examples and are not limiting, and these advantages, benefits and effects and the like cannot be considered as the various embodiments of the present disclosure must have. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and the above details do not limit the present disclosure to the above specific details.

[0111] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between each embodiment can be referred to each other. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0112] The methods and apparatus of the present disclosure can be implemented in a number of ways. For example, the methods and apparatus of the present disclosure can be implemented via software, hardware, firmware, or any combination of software, hardware, and firmware. The above described order of steps for the methods is merely illustrative, and the steps of the methods of the present disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, which includes machine readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the methods according to the present disclosure.

[0113] The foregoing description has been presented for the purposes of illustration and description. Furthermore, the description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, additions, and sub-combinations, which fall within the scope of the embodiments.

Claims

1. A system for implementing remote medical video conferencing, characterized in that, The system includes: at least one first terminal device serving as a near-end participant device in a video conference, at least one second terminal device serving as a far-end participant device in a video conference, at least one controlled device, and at least one first control device. The first terminal device includes: at least one controlled device interface and multiple network access interfaces. The first terminal device is connected to the controlled device through the controlled device interface, and the first terminal device accesses the network through its multiple network access interfaces to form a multi-link transmission channel. The second terminal device includes at least one remote control interface and at least one network access interface. The second terminal device is connected to the first control device through its remote control interface, and the second terminal device is connected to the network through its network access interface to form a link transmission channel. The first terminal device is used to collect multimedia information and near-end multimedia information of the controlled device, allocate a first transmission priority to the multimedia information of the controlled device, allocate a second transmission priority to the near-end multimedia information, and allocate link transmission channels for the multimedia information of the controlled device and the near-end multimedia information respectively according to the data transmission performance of each link transmission channel obtained by monitoring, the first transmission priority, and the second transmission priority, and send the multimedia information of the controlled device and the near-end multimedia information to the second terminal device; wherein, the first transmission priority is higher than the second transmission priority; The second terminal device is used to receive first control information for the controlled device generated by the first control device through its remote control interface, and to collect remote multimedia information, allocate a third transmission priority to the first control information, allocate a fourth transmission priority to the remote multimedia information, and send the first control information and remote multimedia information to the first terminal device through its link transmission channel according to the third transmission priority and the fourth transmission priority; wherein, the third transmission priority is higher than the fourth transmission priority. The first terminal device is also used to receive first control information and remote multimedia information from the second terminal device, and according to the first control information, send a control command to the controlled device through the controlled device interface so that the controlled device performs corresponding operations based on the control command and plays the remote multimedia information in the video conferencing interface; The second terminal device is also used to receive controlled device multimedia information and near-end multimedia information from the first terminal device, and to play the controlled device multimedia information and near-end multimedia information in the video conferencing interface.

2. The system according to claim 1, characterized in that, The first terminal device includes at least one remote control interface. The first terminal device is connected to the second control device through its remote control interface. The first terminal device is further configured to receive second control information for the controlled device generated by the second control device, and send a control command to the controlled device through the controlled device interface according to the second control information, so that the controlled device performs a corresponding operation based on the control command.

3. The system according to claim 1 or 2, characterized in that, The second terminal device includes: multiple network access interfaces, and the second terminal device accesses the network through its multiple network access interfaces to form a multi-link transmission channel; The second terminal device is specifically used to allocate link transmission channels for the first control information and remote multimedia information respectively, based on the data transmission performance of each link transmission channel obtained from monitoring, the third transmission priority, and the fourth transmission priority, and to send the first control information and remote multimedia information to the first terminal device.

4. A method for implementing remote medical video conferencing, characterized in that, The method includes the following steps performed in a first terminal device, which serves as a near-end participant device in a video conference, and the first terminal device includes: at least one controlled device interface, multiple network access interfaces, a first video conferencing module, a first multi-link aggregation transmission module, and a remote control module. The first terminal device is connected to the controlled device through the controlled device interface, and the first terminal device accesses the network through its multiple network access interfaces to form a multi-link transmission channel. The steps performed in the first terminal device, which serves as the near-end participant device in the video conference, include: The first video conferencing module collects multimedia information from the controlled device and near-end multimedia information, assigns a first transmission priority to the multimedia information from the controlled device, assigns a second transmission priority to the multimedia information from the near-end, and transmits the multimedia information from the controlled device, the multimedia information from the near-end, the first transmission priority, and the second transmission priority to the first multi-link aggregation transmission module; wherein, the first transmission priority is higher than the second transmission priority. The first multi-link aggregation transmission module monitors the data transmission performance of the multi-link transmission channels, and allocates link transmission channels for the controlled device multimedia information and near-end multimedia information respectively according to the data transmission performance of each link transmission channel, the first transmission priority and the second transmission priority, and sends the controlled device multimedia information and near-end multimedia information to the second terminal device, which is the remote participant device in the video conference. The first video conferencing module receives first control information and remote multimedia information for the controlled device from the second terminal device through the first multi-link aggregation transmission module, and transmits the first control information to the remote control module to play the remote multimedia information in the video conferencing interface; The remote control module sends control commands to the controlled device through the controlled device interface based on the first control information from the first video conferencing module.

5. The method according to claim 4, characterized in that, The method further includes: The first video conferencing module sets the shared control permission information of the controlled device according to the permission settings of the user with management authority, and verifies the shared control permission of the shared control request according to the shared control permission information of the controlled device when it receives a shared control request from the second terminal device. The first video conferencing module assigns a third transmission priority to the shared control permission verification result and transmits the verification result and the third transmission priority to the first multi-link aggregation transmission module; wherein the third transmission priority is not lower than the first transmission priority; The first multi-link aggregation transmission module allocates a link transmission channel for the verification result based on the data transmission performance of each link transmission channel and the third transmission priority, and sends the verification result to the second terminal device.

6. The method according to claim 5, characterized in that: The step of transmitting the first control information to the remote control module includes: When the first video conferencing module determines, based on the verification result, that the first control information belongs to the shared control permissions of the second terminal device, it transmits the first control information to the remote control module. or The remote control module sends control commands to the controlled device through the controlled device interface based on the first control information from the first video conferencing module, including: When the remote control module determines, based on the verification result, that the first control information belongs to the shared control permissions of the second terminal device, it sends a control command to the controlled device through the controlled device interface based on the first control information.

7. The method according to claim 4, characterized in that, The near-end multimedia information includes at least one of local device multimedia information and current environment multimedia information, and the first terminal device further includes at least one of audio and video acquisition interface and microphone and camera unit, and the first terminal device is connected to at least one local device through the audio and video acquisition interface; The first video conferencing module collects multimedia information from the controlled device and near-end multimedia information, including: The first video conferencing module preprocesses the local device multimedia information and / or current environment multimedia information transmitted through the audio and video acquisition interface and / or microphone and camera unit, and encodes the preprocessed local device multimedia information and / or current environment multimedia information. or The preprocessing unit in the first terminal device preprocesses the local device multimedia information and / or current environment multimedia information transmitted through the audio and video acquisition interface and / or microphone and camera unit. The codec in the first terminal device encodes the preprocessed local device multimedia information and / or current environment multimedia information output by the preprocessing unit. The first video conferencing module receives the encoded information output by the encoding and decoding unit to obtain the collected local device multimedia information and / or current environment multimedia information; The preprocessing includes at least one of noise reduction processing and echo cancellation processing; Assigning a second transmission priority to the near-end multimedia information includes: The first video conferencing module assigns different transmission priorities to the local device multimedia information and / or the current environment multimedia information, and the transmission priority assigned by the first video conferencing module to the local device multimedia information is higher than the transmission priority assigned to the current environment multimedia information.

8. The method according to claim 4, characterized in that, The first terminal device includes at least one remote control interface, the first terminal device being connected to the second control device via its remote control interface, and the method further includes: The first video conferencing module receives second control information for the controlled device generated by the second control device through the remote control interface, and transmits the second control information to the remote control module. The remote control module sends control commands to the controlled device through the controlled device interface based on the second control information from the first video conferencing module.

9. The method according to any one of claims 4 to 8, characterized in that, The method further includes: a step performed in a second terminal device that serves as a remote participant device in a video conference, wherein the second terminal device includes: at least one remote control interface, at least one network access interface, a second video conferencing module, and a link transmission module, wherein the second terminal device is connected to the first control device through the remote control interface, and the second terminal device accesses the network through its network access interface to form a link transmission channel. The steps performed in the second terminal device, which serves as the device of the remote participant in the video conference, include: The second video conferencing module receives first control information for the controlled device generated by the first control device through the remote control interface, collects remote multimedia information, assigns a third transmission priority to the first control information, assigns a fourth transmission priority to the remote multimedia information, and transmits the first control information, remote multimedia information, third transmission priority, and fourth transmission priority to the link transmission module; wherein, the third transmission priority is higher than the fourth transmission priority. The link transmission module sends the first control information and remote multimedia information to the first terminal device through the network access interface according to the third transmission priority and the fourth transmission priority. The second video conferencing module receives controlled device multimedia information and near-end multimedia information from the first terminal device through the link transmission module, and plays the controlled device multimedia information and near-end multimedia information in the video conferencing interface.

10. The method according to claim 9, characterized in that, The second terminal device includes multiple network access interfaces, and the second terminal device accesses the network through its multiple network access interfaces to form a multi-link transmission channel; the link transmission module includes a second multi-link aggregation transmission module; The second multi-link aggregation transmission module monitors the data transmission performance of its multi-link transmission channels, and allocates link transmission channels for the first control information and remote multimedia information respectively according to the data transmission performance of each link transmission channel, the third transmission priority and the fourth transmission priority, and sends the first control information and remote multimedia information to the first terminal device.

11. The method according to claim 10, characterized in that, The step of allocating link transmission channels for the first control information and the remote multimedia information respectively includes: The second multi-link aggregation transmission module allocates basic link transmission channels and redundant link transmission channels for the first control information based on the load balancing principle. The method further includes: When the second multi-link aggregation transmission module determines that the data transmission performance of the link transmission channel currently transmitting the first control information does not meet the transmission performance requirements, it performs a link transmission channel switching operation for the first control information based on the load balancing principle, according to the link transmission channel that currently meets the transmission performance requirements.

12. A computer-readable storage medium storing a computer program for performing the method of any one of claims 4-11.

13. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 4-11.

Images