Operation support platform supporting remote picture synchronous freeze-frame and audio and video interaction method thereof

Abstract

The invention provides an operation support platform supporting remote picture synchronous freeze-frame and an audio and video interaction method thereof. The operation support platform comprises audio and video acquisition equipment arranged in an operating room and configured to perform acquisition and frame extraction on audios and videos in the operating room, and the videos are 4K high-definition videos; the local terminal display equipment is arranged in the operating room and is configured to locally display the synchronously freeze-frame picture after the frame extraction; the audio and video interaction server is arranged at a remote end and is configured to perform compression and transport stream replacement on the synchronous freeze-frame picture after frame extraction; and the remote interaction equipment is arranged at a remote end and is configured to remotely display the synchronous freeze-frame picture after the frame extraction. According to the operation support platform and method, synchronous picture freeze-frame in the remote consultation process is achieved, it is ensured that the two parties observe the completely same picture, continuous collection of background audios and videos is maintained while picture freeze-frame is ensured, and the integrity of operation records is not affected.

Description

Technical Field

[0001] The present invention relates to medical imaging, and in particular to a surgery support platform supporting remote image synchronization and freezing and an audio and video interaction method thereof. Background Art

[0002] In the modern and highly developed medical system, remote consultation in the operating room and expert support are increasingly valued as key measures to improve the efficiency of medical resource allocation and medical level. During the operation, doctors often need to quickly and accurately discuss and analyze some fleeting and critical surgical images in order to adjust the surgical strategy or confirm the surgical effect in time. However, despite its huge potential, the existing remote consultation system still faces a series of technical challenges that need to be solved in practical applications.

[0003] In the prior art, there are also some operating room remote consultation and surgical guidance systems, but these prior arts still have the following problems: it is impossible to achieve the instant freezing function of the picture while ensuring that the entire surgical process is recorded continuously and uninterruptedly in the background for doctors to observe and discuss in detail.

[0004] In addition, when remote experts and medical staff in the operating room use video links to jointly analyze surgical images, due to technical limitations, it is difficult for both parties to ensure that the images they see are completely consistent. This lack of synchronization not only affects the accuracy and efficiency of communication between the two parties, but also easily causes misunderstandings and communication barriers, which may pose a potential threat to the smooth progress of the operation and the safety of the patient. Summary of the invention

[0005] In view of the deficiencies of the prior art, the present invention provides a surgery support platform and method that supports remote image synchronization and freezing.

[0006] The first embodiment of the present invention is a surgical support platform supporting remote screen synchronization and freezing, comprising:

[0007] The audio and video acquisition equipment installed in the operating room is configured to collect and extract frames of audio and video in the operating room. The video is 4K high-definition video;

[0008] A local terminal display device provided in the operating room is configured to locally display the synchronously frozen images after frame extraction;

[0009] The audio and video interactive server located at the remote end is configured to compress and replace the transmission stream of the synchronous frozen images after frame extraction;

[0010] The remote interactive device arranged at the remote end is configured to remotely display the synchronous frozen picture after frame extraction.

[0011] Preferably, it also includes a local storage device located in the operating room.

[0012] While extracting frames, the audio and video acquisition device stores the audio and video collected in the background in a local storage device.

[0013] The second embodiment of the present invention is an audio and video interaction method for an operating room, which is a method for audio and video interaction using the surgical support platform supporting remote screen synchronization freeze of the first embodiment of the present invention, comprising the following steps:

[0014] The freeze function is triggered on the local terminal display device, and the local terminal display device sends a freeze command to the audio and video acquisition device;

[0015] The audio and video acquisition device extracts the current frame from the acquired video stream as a freeze frame, and displays the freeze frame on the local terminal display device;

[0016] The audio and video acquisition device transmits the freeze frame to the audio and video interaction server;

[0017] The audio and video interactive server compresses the original freeze frame, pauses the current video stream transmission, and replaces the transmission stream with the compressed freeze frame;

[0018] The audio and video interaction server transmits the replaced freeze frame to the remote interaction device.

[0019] Alternatively, the audio and video interaction method includes the following steps:

[0020] When the freeze function is triggered on the remote interactive device, the remote interactive device sends a freeze counter-control instruction to the audio and video interactive server;

[0021] The audio and video interactive server transmits the freeze frame reverse control instruction to the audio and video acquisition device;

[0022] The audio and video acquisition device extracts the current frame from the acquired video stream as a freeze frame, and displays the freeze frame on the local terminal display device;

[0023] The audio and video acquisition device transmits the freeze frame to the audio and video interaction server;

[0024] The audio and video interaction server compresses the original freeze frame, pauses the current video stream transmission, and replaces the transmission stream with the compressed freeze frame;

[0025] The audio and video interaction server transmits the replaced freeze-frame to the remote interaction device.

[0026] Preferably, the freeze frame is extracted by directly extracting frames using a graphics card.

[0027] Preferably, double buffering is used for transmission of freeze frames.

[0028] Double buffering includes the front buffer and the back buffer.

[0029] Preferably, the transmission channel during freeze frame transmission is a high priority transmission channel.

[0030] Preferably, the transmission mode during freeze frame transmission is multicast distribution.

[0031] As described above, the present invention provides a surgical support platform and method that supports remote screen synchronization and freezing. Compared with the prior art, it realizes the synchronization and freezing of the screen during the remote consultation process, ensuring that both parties observe exactly the same screen. While ensuring the freezing of the screen, it maintains the continuous collection of background audio and video without affecting the integrity of the surgical record.

[0032] In order to further understand the features and technical contents of the present invention, reference may be made to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a block diagram of a surgery support platform according to a first embodiment of the present invention.

[0034] Figure 2 This is another block diagram of the surgery support platform according to the first embodiment of the present invention.

[0035] Figure 3 This is a flowchart of remote screen synchronization freezing in the second embodiment of the present invention.

[0036] Figure 4 FIG. 4 is a flowchart of picture extraction and transmission in the second embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is an explanation of the embodiments of the present invention through specific embodiments, and those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the concept of the present invention. In addition, it is stated in advance that the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention. In addition, the term "or" used herein may include any one or more combinations of the associated listed items depending on the actual situation.

[0038] Reference Figure 1 ,in Figure 1This is a block diagram of a surgery support platform according to a first embodiment of the present invention.

[0039] like Figure 1 As shown, the surgical support platform of the first embodiment of the present invention includes an audio and video acquisition device 100, a local terminal display device 200, an audio and video interaction server 300, and a remote interaction device 400. Figure 2 As shown, the surgery support platform of the first embodiment of the present invention preferably further includes a local storage device 500 .

[0040] Specifically, the audio and video acquisition device 100 is configured to acquire and extract frames of audio and video in the operating room, and the video is a 4K high-definition video. 4K high-definition video refers to a video with a resolution of 3840×2160 or 4096×2160.

[0041] The local terminal display device 200 is configured to locally display the synchronized and frozen images after frame extraction, and to display the synchronized and frozen surgical images in real time in the operating room.

[0042] The audio and video interaction server 300 is configured to compress and replace the transmission stream of the synchronous frozen images after frame extraction, and is responsible for the processing and transmission of the audio and video streams.

[0043] The remote interactive device 400 is configured to remotely display the synchronized and frozen images after frame extraction, and to display the synchronized and frozen surgical images to the experts in real time at the remote end.

[0044] Preferably, the audio and video acquisition device 100 stores the audio and video acquired in the background in the local storage device 500 while extracting frames.

[0045] The second embodiment of the present invention is an audio and video interaction method for an operating room, which is a method for audio and video interaction using the surgical support platform supporting remote screen synchronization freeze of the first embodiment of the present invention. The audio and video interaction method corresponds to the surgical support platform of the first embodiment, so the various variations of the first embodiment are also applicable to the second embodiment, which will not be described in detail here.

[0046] Specifically, the audio and video interaction method includes the following steps.

[0047] like Figure 3 As shown in the upper part of the figure, the freeze function is first triggered on the local terminal display device (i.e., the local device in the figure), and the freeze function is triggered by a specific operation instruction or button click, and a freeze instruction is sent. In response to this freeze instruction, the audio and video acquisition device performs the following operations:

[0048] First, accurately extract the current frame when the freeze is triggered from the continuous 4K high-definition video stream.

[0049] Then, the current frame at the time of freezing is displayed as a frozen frame on the user's local terminal screen for immediate viewing and confirmation by the user. The device does not stop working, but continues to collect and store subsequent audio and video streams in the background for possible subsequent needs. Subsequently, this frozen frame is transmitted to the audio and video interactive server (i.e., the interactive server in the figure) via a local area network or other transmission methods.

[0050] After receiving the freeze frame, the audio and video interaction server immediately starts the picture processing flow and performs the following operations:

[0051] First, the original 4K resolution frame images are compressed in an advanced way to reduce the file size for fast transmission and storage. Then the server pauses the current real-time transmission of the video stream to ensure that the freeze frame can be seamlessly inserted, and the compressed freeze frame is accurately replaced with the original video transmission stream to ensure that the remote receiving end can receive the freeze frame.

[0052] After the audio and video interaction server completes processing, the freeze-frame is transmitted to the remote interaction device (i.e., the interaction server in the figure) through multiple network channels (including but not limited to local area network, Internet or high-speed 5G network), allowing remote experts in different locations to share this key freeze-frame instantly.

[0053] like Figure 3 As shown in the lower half of , the audio and video interaction method may also include the following steps.

[0054] First, the freeze function is triggered on the remote interactive device, and a freeze counter-control instruction is sent through a specific operation instruction or button click. In response to this freeze counter-control instruction, the audio and video interactive server transmits the freeze counter-control instruction to the audio and video acquisition device, and the audio and video acquisition device performs the following operations:

[0055] First, accurately extract the current frame when the freeze is triggered from the continuous 4K high-definition video stream.

[0056] Then, the current frame at the time of freezing is displayed as a frozen frame on the user's local terminal screen for immediate viewing and confirmation by the user. The device does not stop working, but continues to collect and store subsequent audio and video streams in the background for possible subsequent needs. Subsequently, this frozen frame is transmitted to the audio and video interactive server (i.e., the interactive server in the figure) via a local area network or other transmission methods.

[0057] After receiving the freeze frame, the audio and video interaction server immediately starts the picture processing flow and performs the following operations:

[0058] First, the original 4K resolution frame images are compressed in an advanced way to reduce the file size for fast transmission and storage. Then the server pauses the current real-time transmission of the video stream to ensure that the freeze frame can be seamlessly inserted, and the compressed freeze frame is accurately replaced with the original video transmission stream to ensure that the remote receiving end can receive the freeze frame.

[0059] After the audio and video interaction server completes processing, the freeze-frame is transmitted to the remote interaction device (i.e., the interaction server in the figure) through multiple network channels (including but not limited to local area network, Internet or high-speed 5G network), allowing remote experts in different locations to share this key freeze-frame instantly.

[0060] like Figure 4 As shown, it is preferred to use the method of directly extracting frames from video memory when extracting frames. The direct video memory reading mechanism accelerated by GPU hardware is adopted to optimize the data transmission path to ensure that the reading task of the current frame is completed efficiently and accurately within milliseconds, thereby greatly reducing the delay of frame processing. By directly obtaining the current frame at the video memory level, the delay problem caused by the transfer of data between different storage media in the traditional method can be avoided, further improving the real-time and smoothness of frame processing. The specific process of direct video memory frame extraction technology based on DirectX is as follows:

[0061] (1) Equipment initialization phase

[0062] First, create a DirectX device, which acts as a bridge to achieve direct, high-speed communication between the computer and the graphics card, providing a basis for subsequent operations.

[0063] Secondly, the DXGI (DirectX Graphics Infrastructure) interface of the graphics card is obtained through the DirectX device. This interface provides direct access to the graphics card's video memory, allowing the system to efficiently read data in the video memory.

[0064] Finally, a dedicated frame buffer is created in the video memory space of the graphics card to store the video frame data directly extracted from the video memory to ensure the integrity and security of the data.

[0065] (2) Frame extraction execution stage

[0066] Continuously monitor new video frame signals through the DirectX interface to ensure that every frame can be captured in real time.

[0067] Once a new frame signal is obtained, the system immediately locks the area in the video memory corresponding to the frame to prevent other operations from interfering with the data in this area.

[0068] Then, the direct access mechanism of video memory provided by DirectX is used to quickly read the frame data in the locked area and temporarily store it in the frame extraction buffer created previously.

[0069] After completing the data reading, the system promptly unlocks the video memory area so that the graphics card can continue to process subsequent frame data.

[0070] (3) Data output stage

[0071] Map the frame data in the frame extraction buffer to the system memory space so that it can be directly accessed and processed by the upper-layer application.

[0072] Provide a concise and efficient frame data access interface for upper-level applications to facilitate subsequent image analysis, processing or display operations.

[0073] When the upper-layer application completes the use of the frame data, the system promptly releases the memory mapping, releases the corresponding system resources, and prepares for subsequent frame data processing.

[0074] like Figure 4 As shown, it is preferred to use a double buffering method to extract the freeze frame, and the double buffering includes a foreground buffer and a background buffer.

[0075] Front buffer: This area is dedicated to local real-time display needs. It is based on an efficient memory operation mechanism and can lock the local image in a very short time, such as within 10 milliseconds (ms). This fast response capability ensures that the image seen locally is smooth and coherent.

[0076] Background buffering: Different from foreground buffering, background buffering is mainly used to prepare for network transmission. Relying on 10G fiber switches and high-speed Internet, it can transmit image data to the terminal in a relatively long but still fast time, such as 200 milliseconds (ms), and successfully display it. It not only ensures the efficiency of network transmission, but also ensures that the image quality is not damaged. The double buffer switching mechanism can eliminate the image tearing phenomenon, ensuring that both local users and remote experts can see synchronized and consistent images. This mechanism achieves the consistency and stability of image display by intelligently managing the switching of foreground and background buffers. The switching process is as follows.

[0077] When the freeze trigger is triggered, the foreground buffer is processed as follows: when the system receives the freeze trigger signal, the foreground buffer will immediately lock the current frame to ensure that the content of this frame will not be overwritten or changed by subsequent images. Then, this locked frame will be quickly pushed to the local display device for users to view in real time. Due to the efficient processing mechanism of the foreground buffer, this process can be completed within 10ms, ensuring the immediacy and stability of the local image.

[0078] The back buffer process is that at the same time as the foreground buffer, the back buffer will also synchronously copy the content of the current frame. Then, the back buffer will optimize the network transmission of this frame, such as compression and encoding, to reduce the data transmission load and increase the transmission speed. Finally, the picture data of this frame is transmitted to the remote device through a high-speed network (such as a 10G fiber optic switch and high-speed Internet).

[0079] During remote presentation, in terms of local presentation, the system directly reads the locked frame from the foreground buffer and continuously displays it on the screen, ensuring that the picture the user sees is clear and stable. For remote presentation, the system first waits for the remote device to confirm receipt of the picture data. Once the confirmation signal is received, the system will synchronously display the optimized frame in the background buffer to the remote device. Due to the efficient processing of the background buffer and the transmission capacity of the high-speed network, this process can be completed within 200ms, ensuring that the remote user can also see the same and synchronized picture as the local one.

[0080] like Figure 4 As shown, the transmission channel during freeze frame transmission is preferably a high priority transmission channel.

[0081] In order to ensure the efficiency and stability of data transmission, a 10G fiber optic switch is used to establish a dedicated high-priority transmission channel, which not only provides extremely high bandwidth, but also ensures the priority transmission of critical data through priority setting, thereby greatly reducing transmission delays.

[0082] In terms of protocol selection, the efficiency of the UDP protocol and the reliability of a simple verification mechanism are combined. The UDP protocol is very suitable for real-time data transmission scenarios with its low latency and high throughput. At the same time, by introducing a simple verification mechanism, errors in data transmission can be detected and corrected to a certain extent, further improving the reliability of transmission.

[0083] like Figure 4 As shown, the preferred transmission mode for freeze frame transmission is multicast distribution.

[0084] In order to meet the needs of multiple clients to receive images at the same time, the server uses multicast technology, which allows the server to send the same data packet to multiple clients at the same time, thereby greatly improving the efficiency of data transmission.

[0085] In order to ensure the transmission quality, the reliable UDP protocol was selected. Compared with the standard UDP protocol, the reliable UDP protocol improves the reliability and stability of data transmission by introducing strategies such as retransmission mechanism and flow control.

[0086] In order to solve the problem of packet loss, a fast retransmission strategy is adopted. When a packet loss is detected, the server will immediately resend the lost packet to ensure that the client can receive complete and continuous picture data, further improving the reliability of data transmission and user experience.

[0087] (Example)

[0088] The interactive mode of the audio and video interaction supporting the remote picture synchronization freezing of the present invention is adopted.

[0089] In a tense scenario of a remote surgical consultation: when the operation reaches the most critical and delicate part, the surgeon can rely on his rich experience and keen insight to quickly judge and decisively click the eye-catching "screen lock" button on the local terminal to instantly freeze the key picture in the current surgical field of view.

[0090] At the same time, with the help of efficient network transmission technology, remote experts also received the frozen images almost at the same time. They were able to observe every detail of the surgical site in detail without being restricted by time and space, thus providing more accurate and professional consultation opinions. This instant information sharing and collaborative work greatly improved the success rate and safety of the surgery.

[0091] During this process, the recording work of the system background was not interrupted, and every step and detail of the operation was still recorded, ensuring the integrity of the entire operation process. This recording not only provides a basis for post-operative review and summary, but also accumulates valuable information for future teaching and scientific research.

[0092] As described above, the audio and video interaction technology for operating rooms that supports remote picture synchronization and freezing provided by the present invention realizes picture synchronization and freezing during remote consultation, ensuring that both parties observe exactly the same picture. While ensuring the picture freezing, the continuous collection of background audio and video is maintained without affecting the integrity of the surgical records.

[0093] Industrial Applicability

[0094] In summary, the present invention provides a surgical support platform and method that supports remote screen synchronization and freezing. Compared with the prior art, it realizes the synchronization and freezing of screens during remote consultation, ensuring that both parties observe exactly the same screen. While ensuring the freezing of the screen, it maintains the continuous collection of background audio and video without affecting the integrity of the surgical record.

[0095] The contents disclosed above are only preferred feasible embodiments of the present invention, and are not intended to limit the scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention description and drawings are included in the scope of the present invention.

Claims

1. A surgical support platform supporting remote image synchronization and freezing, characterized in that: Include: The audio and video acquisition equipment installed in the operating room is configured to collect and extract frames of audio and video in the operating room. The video is 4K high-definition video; A local terminal display device provided in the operating room is configured to locally display the synchronously frozen images after frame extraction; The audio and video interactive server located at the remote end is configured to compress and replace the transmission stream of the synchronous frozen images after frame extraction; The remote interactive device arranged at the remote end is configured to remotely display the synchronous frozen picture after frame extraction.

2. The surgical support platform according to claim 1, wherein: It also includes local storage devices located in the operating room. While extracting frames, the audio and video acquisition device stores the audio and video collected in the background in a local storage device.

3. An audio and video interaction method supporting remote image synchronization and freezing, characterized in that: A method for audio and video interaction using the surgical support platform as claimed in claim 1 or 2, comprising the following steps: The freeze function is triggered on the local terminal display device, and the local terminal display device sends a freeze command to the audio and video acquisition device; The audio and video acquisition device extracts the current frame from the acquired video stream as a freeze frame, and displays the freeze frame on the local terminal display device; The audio and video acquisition device transmits the freeze frame to the audio and video interaction server; The audio and video interactive server compresses the original freeze frame, pauses the current video stream transmission, and replaces the transmission stream with the compressed freeze frame; The audio and video interaction server transmits the replaced freeze frame to the remote interaction device. Alternatively, the audio and video interaction method includes the following steps: When the freeze function is triggered on the remote interactive device, the remote interactive device sends a freeze counter-control instruction to the audio and video interactive server; The audio and video interactive server transmits the freeze frame reverse control instruction to the audio and video acquisition device; The audio and video acquisition device extracts the current frame from the acquired video stream as a freeze frame, and displays the freeze frame on the local terminal display device; The audio and video acquisition device transmits the freeze frame to the audio and video interaction server; The audio and video interactive server compresses the original freeze frame, pauses the current video stream transmission, and replaces the transmission stream with the compressed freeze frame; The audio and video interaction server transmits the replaced freeze-frame to the remote interaction device.

4. The audio and video interaction method according to claim 3, characterized in that: Use the graphics card to directly extract frames to extract freeze frames.

5. The audio and video interaction method according to claim 3, characterized in that: Use double buffering to transmit freeze frames. Double buffering includes the front buffer and the back buffer.

6. The audio and video interaction method according to claim 3, characterized in that: The transmission channel during freeze frame transmission is a high priority transmission channel.

7. The audio and video interaction method according to claim 3, characterized in that: The transmission mode for freeze frame transmission is multicast distribution.

Images