Operation support platform based on audio and video interaction and remote interaction method thereof

Abstract

The invention provides an operation support platform based on audio and video interaction and a remote interaction method thereof. The operation support platform comprises: an audio control module configured to perform acquisition and privacy protection on audio in an operating room; the remote interaction module is configured to provide remote interaction between medical staff in the operating room and remote experts outside the operating room; the picture combination module is configured to combine pictures of videos shot by a shooting device in the operating room and then display the combined pictures; and the storage management module is configured to manage storage data in the whole system. According to the operation support platform and method provided by the invention, the audio privacy protection capability in remote interaction is remarkably improved, the remote interaction performance is remarkably improved, and the remote guidance experience is remarkably improved.

Description

Technical Field

[0001] The present invention relates to medical imaging, and in particular to a surgery support platform based on audio and video interaction and a remote interaction method thereof. Background Art

[0002] In the modern highly developed medical system, remote consultation in operating rooms and expert support are increasingly valued as key measures to improve the efficiency of medical resource allocation and medical level. However, despite its huge potential, the existing remote consultation system still faces a series of technical challenges that need to be solved in practical application.

[0003] In the prior art, there are also some operating room remote consultation and surgery guidance systems, but these prior arts still have the following problems:

[0004] 1. Insufficient audio privacy protection:

[0005] Unable to quickly cut off audio transmission in an emergency: When an emergency occurs during surgery or when patient privacy needs to be urgently protected, the system lacks a mechanism to quickly interrupt audio transmission, which may lead to the leakage of sensitive information.

[0006] The privacy protection needs of all parties are difficult to unify and coordinate: the privacy protection needs of all parties involved in the consultation (such as patients, operating room medical staff, remote experts, etc.) are different, and it is difficult to find a balance in the system design to meet the requirements of all parties.

[0007] Blurred privacy boundaries between remote experts and operating rooms: During remote consultations, the flow of information between experts and operating rooms lacks clear privacy boundaries, increasing the risk of privacy leaks.

[0008] 2. Poor remote interaction experience:

[0009] Limited guidance marking function: The marking tools provided by existing systems often have limited functions and cannot meet the precise guidance needs in complex surgical scenarios, which affects the guidance efficiency of remote experts.

[0010] Insufficient support for multi-party consultation: When multiple experts are required to participate in the consultation, the system's support for multi-party interaction is not perfect, which may lead to poor communication or information omission.

[0011] The interactive method is not intuitive enough: Existing remote interactive methods often rely too much on text or simple graphic marks and lack intuitiveness, making it difficult for remote experts to accurately understand the actual situation in the operating room, thus affecting the consultation effect. Summary of the invention

[0012] In view of the deficiencies of the prior art, the present invention provides a surgical support platform and method based on audio and video interaction.

[0013] The first embodiment of the present invention is a surgical support platform based on audio and video interaction, comprising:

[0014] an audio control module configured to collect and protect the privacy of audio in the operating room;

[0015] A remote interaction module configured to provide remote interaction between medical staff in the operating room and remote experts outside the operating room;

[0016] A picture combination module, configured to combine and display pictures of the video captured by the camera in the operating room;

[0017] The storage management module is configured to manage storage data in the entire system.

[0018] Preferably, the audio control module adopts a multi-level audio control strategy to synchronize the audio status.

[0019] Preferably, the multi-level audio control strategy includes:

[0020] Audio access permission management based on RBAC;

[0021] Independent audio control for different zones;

[0022] Selective listening control for remote experts.

[0023] Preferably, the state of the audio is kept synchronized in a publish / subscribe model.

[0024] Preferably, the remote interaction module includes a transmission optimization unit for reducing the delay of the remote interaction.

[0025] Preferably, the remote interaction module includes an electronic marking unit for real-time marking.

[0026] Preferably, the remote interaction module includes an interaction management unit for managing the rights of the remote interaction.

[0027] Preferably, the picture combination module performs 2x2 or 3x3 picture combination.

[0028] The second embodiment of the present invention is a remote interaction method for an operating room, which is a method for performing audio and video interaction using the surgery support platform based on audio and video interaction of the first embodiment of the present invention.

[0029] As described above, the present invention provides a surgical support platform and method based on audio and video interaction, which significantly improves the audio privacy protection capability in remote interaction, significantly improves the remote interaction performance, and significantly improves the remote guidance experience compared with the prior art.

[0030] In order to further understand the features and technical contents of the present invention, reference may be made to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a block diagram of the surgical support platform according to the first embodiment of the present invention when applied to remote interaction during surgery.

[0032] Figure 2 FIG. 4 is a flowchart of audio control in the first embodiment of the present invention.

[0033] Figure 3 It is a schematic diagram of the remote interaction mechanism in the first embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following is an explanation of the embodiments of the present invention through specific embodiments, and those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the concept of the present invention. In addition, it is stated in advance that the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention. In addition, the term "or" used herein may include any one or more combinations of the associated listed items depending on the actual situation.

[0035] Reference Figure 1 ,in Figure 1 This is a block diagram of the surgical support platform according to the first embodiment of the present invention when applied to remote interaction during surgery.

[0036] like Figure 1 As shown, the surgical support platform based on audio and video interaction of the first embodiment of the present invention (in Figure 1 When the audio and video forwarding server is used for remote interaction during surgery, it is connected to the operating room terminal and the remote expert workstation. The audio and video forwarding server includes an audio control module 100, a remote interaction module 200, a screen combination module 300, and a storage management module 400. The operating room terminal is provided with a camera and a microphone. The remote interaction control interface is displayed on the remote expert workstation.

[0037] Specifically, the audio control module 100 is configured to collect and protect the privacy of audio in the operating room. The audio control module 100 uses the following structure to protect the privacy of audio.

[0038] Emergency Audio Control Unit: When a potential risk of privacy leakage is detected, it can be quickly activated and automatically cut off audio transmission to ensure user privacy security.

[0039] Quick audio off control: You can turn off audio collection with one click through the intuitive button on the system's software interface. The operation is simple and quick, and effectively protects personal privacy.

[0040] Independent audio control unit at the remote expert end: Remote experts at the remote expert workstation can independently control the audio acquisition and transmission at their end to achieve flexible and secure remote collaboration.

[0041] Audio transmission cut-off response unit: Through an efficient cut-off mechanism, the system can cut off audio transmission within milliseconds to ensure that the risk of privacy leakage is minimized.

[0042] The audio control module 100 also uses the following structure to quickly respond to the information transmission layer.

[0043] Real-time Audio Transmission Protocol Department: Based on the UDP protocol, the real-time and reliability of audio data transmission are optimized to ensure high-quality transmission of audio data.

[0044] Priority flag: By adding a priority flag to the audio data packet, the system can intelligently identify and prioritize the transmission of key audio data, thereby improving transmission efficiency.

[0045] Independent control channel transmission department: Transmit emergency control instructions through a dedicated channel to ensure the timeliness and accuracy of the instructions and reduce the risk of delays.

[0046] Control instruction part: Contains global timestamp for state synchronization. The global timestamp is embedded in the control instruction, which helps the system achieve accurate state synchronization and ensure the consistency of audio transmission and control.

[0047] The audio control module 100 also uses the following structure to quickly respond to the processing of the server side.

[0048] QoS mechanism: Through the QoS (Quality of Service) mechanism, the server can intelligently manage network resources and bandwidth to ensure the stability and reliability of audio transmission. QoS provides different priorities for different users or different data streams, and ensures that the performance of the data stream reaches a certain level according to the requirements of the application.

[0049] Dedicated bandwidth department: Dedicated bandwidth is reserved to ensure timely transmission of control instructions. Dedicated bandwidth is reserved on the server side for priority transmission of control instructions to ensure timely response and execution of instructions.

[0050] The audio control module 100 also uses the following structure to quickly respond to the client's processing.

[0051] Asynchronous processing framework: It adopts an efficient event-driven framework that can respond to various control instructions in real time to ensure the flexibility and controllability of audio acquisition and transmission.

[0052] Independent control instruction processing unit: The processing of control instructions is separated from the audio and video decoding thread to ensure the timeliness of instruction processing and the smoothness of audio and video decoding.

[0053] The audio control module 100 preferably adopts the following multi-level audio control strategy, but is not limited thereto, and the control strategy may be adjusted as needed.

[0054] 1. RBAC-based audio access rights management: The system adopts the RBAC (Role-based access control) model to achieve refined management of audio access rights and ensure secure access to audio resources.

[0055] 2. Support independent audio control of different areas such as surgical area and consultation area: The system supports independent control of audio in different areas to meet the needs of multiple scenarios.

[0056] 3. Realize selective listening control of remote experts: Remote experts can selectively listen to audio in different areas according to actual needs, thereby improving the efficiency and flexibility of remote collaboration.

[0057] The audio control module 100 preferably adopts the following audio state synchronization mechanism, but is not limited thereto, and the synchronization mechanism may be adjusted as needed.

[0058] 1. Real-time state synchronization using the publish-subscribe model: The system uses the publish-subscribe model to achieve real-time synchronization of audio state and ensure the consistency of audio state on each end.

[0059] 2. Provide audio control log recording and auditing: The system records and saves audio control logs to facilitate subsequent auditing and tracing, ensuring the security and compliance of audio control operations.

[0060] like Figure 2As shown, an exemplary example of the process of audio control of the audio control module 100 is that after the control starts, the emergency audio control is triggered, and then the local audio collection is stopped within a delay of less than 1ms, and then the control command is sent within a delay of less than 5ms, and then the server is broadcast within a delay of less than 10ms, and the update delay on the remote expert workstation is less than 20ms. The delay times in the above examples can also be adjusted as needed. When the audio collection stops, an event-driven asynchronous processing framework is adopted, and the control instruction processing is independent of the audio and video decoding thread, and the API of the underlying audio acquisition device is directly operated. When sending the control command, a custom real-time audio transmission protocol is implemented based on the UDP protocol, and the emergency control instruction is transmitted using an independent control channel. When broadcasting, the quality of service (QoS) mechanism is used to adopt the real-time state synchronization of the publish-subscribe mode. When updating on the remote expert workstation, an event-driven asynchronous processing framework is adopted, and the control instruction processing is independent of the audio and video decoding thread, and the real-time state synchronization adopts the publish-subscribe mode.

[0061] Specifically, the remote interaction module 200 is configured to provide remote interaction between medical staff in the operating room and remote experts outside the operating room. The remote interaction module 200 preferably includes a transmission optimization unit for reducing the delay of remote interaction. For example, the WebRTC (Web Real-Time Communication) standard can be used to achieve end-to-end efficient and secure transmission to ensure the real-time and stability of data during transmission. An intelligent bandwidth adaptation strategy can also be used to dynamically adjust the transmission rate according to network conditions to avoid freezes and delays and improve user experience.

[0062] The remote interaction module 200 preferably includes an electronic marking unit for real-time marking. The electronic marking unit supports real-time marking of electronic handwriting, which can be directly and freely drawn and annotated to achieve more intuitive communication and collaboration. In addition, it can also provide high-quality voice interaction function to ensure that multiple participants can communicate clearly and smoothly through voice, thereby enhancing interactivity.

[0063] The remote interaction module 200 preferably includes an interaction management unit for managing the rights of remote interaction. The role-based permission control mechanism ensures that participants have corresponding access and operation rights according to their roles, ensuring security and order. It also provides multi-party consultation room management functions and conference recording and playback functions, allowing the recording of conference content for subsequent viewing and review, and supports playback functions for convenient review of conference details at any time.

[0064] like Figure 3As shown in the figure, during remote interaction, the interface layout is: the left side is the real-time screen display area of ​​the operating room, and the right side is the remote expert control interface, which includes: surgical screen display area, marking toolbar, audio and video control bar. The marking toolbar includes: a brush tool that supports free drawing of surgical guidance lines, a text tool that can add text descriptions and annotations, an eraser tool that removes added marking content, and provides a variety of color options to meet the color selection of different marking needs. The audio and video control bar includes: one-click mute and interactive switch. One-click mute provides fast audio cut-off capability, supports instant response in emergency situations and visual display of mute status. The interactive switch controls the on / off of the remote interactive function, manages the use rights of the marking tool, and provides real-time feedback on the interactive status.

[0065] During remote interaction, the operating room screen and remote marks are synchronized in both directions for two-way real-time transmission; the remote expert's marked content is displayed in real time in the operating room for mark synchronization; the audio and video status of each end is displayed synchronously in real time for status synchronization.

[0066] In this way, the real-time nature of marking and control instructions is ensured, and low-latency transmission is achieved; accurate superposition display of marking content is supported, and precise positioning is achieved; operation status and control effects are immediately visible, and real-time feedback is achieved. During the interaction process, remote experts can view the real-time screen of the operating room and use marking tools for surgical guidance. The marking content will be synchronized to the operating room display in real time, and the audio interaction status can be controlled as needed. Figure 3 As shown, the overall interactive interface is simple and intuitive, the control interface has a clear layout, and the operation is simple; the functions are concentrated, and the commonly used functions are displayed together to improve operating efficiency; the status is clear, and the status of each function is clearly visible to avoid misoperation.

[0067] The picture combination module 300 is configured to combine and display the pictures of the videos taken by the shooting device in the operating room, preferably a 2x2 or 3x3 picture combination, and can also dynamically adjust the combination layout, and can also adjust the display quality according to network adaptation.

[0068] The storage management module 400 is configured to manage storage data in the entire system.

[0069] The second embodiment of the present invention is a remote interaction method for an operating room, which is a method for audio and video interaction using the first embodiment of the present invention, which is a surgical support platform based on audio and video interaction. The remote interaction method corresponds to the surgical support platform of the first embodiment, so the various variations of the first embodiment are also applicable to the second embodiment, which will not be described in detail here.

[0070] The following describes the audio and video interaction supporting remote interaction of the present invention through a specific embodiment.

[0071] (Example)

[0072] The emergency audio control is performed by the audio and video interaction supporting remote interaction of the present invention.

[0073] When an emergency occurs, operating room staff first quickly turn off the audio by clicking the emergency mute icon on the software interface or using a preset shortcut key combination.

[0074] The system then responds as follows: When the system detects an emergency mute command, the local audio capture device will immediately stop all audio capture activities to ensure that sensitive or inappropriate audio content is not transmitted. Then, the system will quickly send an emergency mute command to the audio and video forwarding server, requiring the server to immediately interrupt the transmission of the current audio stream. After receiving the command, the server will immediately broadcast an audio status change message to all relevant clients to ensure that all participants can understand the current audio status in real time. The client on the remote expert side will update its audio status display in real time based on the received status change message, and can intuitively see that the current audio has been muted.

[0075] When audio recovery is required, the following process is performed: After confirming that the emergency situation has been resolved, the system will not immediately restore audio transmission, but wait for the user with recovery authority to authorize the operation. After the user authorizes, the system will gradually restore the audio transmission function according to the preset step-by-step recovery strategy to ensure a smooth and orderly recovery process. During the entire recovery process, the system will automatically record the operation log of each step for subsequent review or troubleshooting.

[0076] Emergency audio control needs to meet the following technical indicators:

[0077] 1. The audio shutdown response time is strictly controlled within 50 milliseconds to ensure that audio transmission can be quickly cut off in an emergency.

[0078] 2. The status synchronization delay is less than 100 milliseconds, ensuring that all clients can understand the current audio status in real time.

[0079] 3. The system supports concurrent control of up to 8 parties to meet the needs of complex scenarios.

[0080] 4. The retention period of operation logs can be configured according to actual needs to ensure the availability and security of log information.

[0081] As described above, through the audio and video interaction technology for operating rooms that supports remote interaction provided by the present invention, good audio privacy protection is achieved, the audio status in multi-party consultation scenarios can be synchronized, low-latency remote interaction is achieved, information transmission is optimized, and synchronous marking can be achieved. Electronic handwriting processing is supported, and a remote marking and guidance system is realized.

[0082] Industrial Applicability

[0083] In summary, the present invention provides a surgical support platform and method based on audio and video interaction. Compared with the prior art, the audio privacy protection capability in remote interaction is significantly improved, the remote interaction performance is significantly improved, and the remote guidance experience is significantly improved.

[0084] The contents disclosed above are only preferred feasible embodiments of the present invention, and are not intended to limit the scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention description and drawings are included in the scope of the present invention.

Claims

1. A surgical support platform based on audio and video interaction, characterized in that: Include: an audio control module configured to collect and protect the privacy of audio in the operating room; A remote interaction module configured to provide remote interaction between medical staff in the operating room and remote experts outside the operating room; A picture combination module, configured to combine and display pictures of the video captured by the camera in the operating room; The storage management module is configured to manage storage data in the entire system.

2. The surgical support platform according to claim 1, wherein: The audio control module adopts a multi-level audio control strategy to synchronize the audio status.

3. The surgical support platform according to claim 2, wherein: The multi-level audio control strategy includes: Audio access permission management based on RBAC; Independent audio control for different zones; Selective listening control for remote experts.

4. The surgical support platform according to claim 2, wherein: The state of the audio is kept synchronized in a publish / subscribe fashion.

5. The surgical support platform according to claim 1, wherein: The remote interaction module includes a transmission optimization unit for reducing the delay of the remote interaction.

6. The surgical support platform according to claim 1, wherein: The remote interaction module includes an electronic marking unit for performing real-time marking.

7. The surgical support platform according to claim 1, wherein: The remote interaction module includes an interaction management unit for managing the rights of remote interaction.

8. The surgical support platform according to claim 1, wherein: The picture combination module performs 2x2 or 3x3 picture combination.

9. A remote interaction method for an operating room supporting remote interaction, characterized in that: A method for audio and video interaction using the surgical support platform as described in any one of claims 1 to 8.

Images