Multi-path medical video acquisition and combined display system and method based on time synchronization

Abstract

The invention provides a multi-path medical video acquisition and combined display system and method based on time synchronization. The multi-channel medical video acquisition and combined display system comprises a signal acquisition part configured to acquire a plurality of audio and video signals from an input device; the signal processing part is configured to convert the plurality of audio and video signals acquired by the signal acquisition part; the core processing part is configured to perform core processing on the plurality of audio and video signals converted by the signal processing part to generate a display picture, and the core processing comprises synchronization, combination and timestamp injection; and a display output unit configured to display and output the display screen generated by the core processing unit. According to the multi-channel medical video acquisition and combined display system and method, the problem of unified management of multi-device video acquisition is solved, accurate time synchronization of multi-channel videos is realized, the display effect is remarkably improved, and time information management is enhanced.

Description

Technical Field

[0001] The present invention relates to medical imaging, and in particular to a multi-channel medical video acquisition and combined display system and method based on time synchronization. Background Art

[0002] The operating room is an important place for patients to undergo surgical treatment and rescue, involving the participation of medical personnel from various fields, and it is extremely important to provide patients with high-quality care. With the increasing complexity of modern surgery, especially the birth of minimally invasive surgery, the requirements for the medical interaction system in the operating room are extremely high.

[0003] In the prior art, there are also some interactive systems for medical use, but these prior arts still have the following problems:

[0004] 1. During the operation, the medical team needs to rely on a variety of high-tech medical equipment, including but not limited to surgical field cameras, panoramic cameras, C-arms, O-arms, endoscopes, laparoscopy, and microscopes. However, these devices operate independently, and the image data formats they generate are diverse, and the interface standards are also different, which brings great challenges to unified collection and management. Due to the inconsistency of data formats, it is difficult for the medical team to efficiently integrate image data from different devices, which affects the making of surgical decisions and the improvement of surgical efficiency.

[0005] 2. Even more difficult is that the image data generated by various medical devices lack an effective synchronization mechanism on the timeline. This leads to a series of problems: first, it is impossible to accurately restore the complete process of the operation, because the images recorded by different devices may be misplaced at the time point; second, there is a significant time difference between the images of multiple devices, making it difficult for doctors to form a coherent cognition when reviewing the operation process; finally, this time inconsistency seriously affects the accuracy and completeness of the operation record, which brings obstacles to the improvement of medical quality.

[0006] 3. In terms of processing and displaying multi-channel video signals, traditional systems also expose many deficiencies. Specifically, the large delay in image combination is a significant problem, which will cause doctors to be unable to obtain clear image information in real time during surgery; at the same time, the unsatisfactory display effect is also a problem that cannot be ignored, because the blurred image quality will interfere with the doctor's judgment; in addition, the traditional system lacks flexible layout adjustment capabilities and cannot meet the doctor's diverse needs for image layout in different surgical scenarios. These problems jointly restrict the effective use of medical imaging information in the operating room. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a multi-channel medical video acquisition and combined display system and method based on time synchronization.

[0008] The first embodiment of the present invention is a multi-channel medical video acquisition and combined display system based on time synchronization, comprising:

[0009] A signal acquisition unit configured to acquire a plurality of audio and video signals from an input device;

[0010] A signal processing unit configured to convert a plurality of audio and video signals collected by the signal collection unit;

[0011] The core processing unit is configured to generate a display screen after performing core processing on the multiple audio and video signals converted by the signal processing unit, and the core processing includes: synchronization, combination, and time stamp injection;

[0012] The display output unit is configured to display and output the display screen generated by the core processing unit.

[0013] Preferably, the interfaces supported by the signal acquisition unit include: HDMI, SDI, DVI, and VGA.

[0014] Preferably, the signal processing unit includes a signal format conversion module and a color space conversion module.

[0015] Preferably, the core processing unit includes: a time synchronization processing module for synchronization, a picture combination module for combination, and a timestamp injection module for timestamp injection.

[0016] Preferably, the synchronization processing module performs synchronization by setting a buffer queue.

[0017] Preferably, the picture combination module uses GPU acceleration to perform data operations in a zero-copy manner and adaptively adjusts the frame rate.

[0018] Preferably, the picture combination module performs 2x2 or 3x3 picture combination.

[0019] Preferably, the display output unit also includes a storage and recording module.

[0020] The second embodiment of the present invention is an audio and video interaction method, which is a method for audio and video interaction using the multi-channel medical video acquisition and combined display system based on time synchronization of the first embodiment of the present invention.

[0021] As described above, the present invention provides a multi-channel medical video acquisition and combined display system and method based on time synchronization. Compared with the prior art, it solves the problem of unified management of multi-device video acquisition, realizes accurate time synchronization of multi-channel videos, significantly improves the display effect, and enhances time information management.

[0022] In order to further understand the features and technical contents of the present invention, reference may be made to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a block diagram of the multi-channel medical video acquisition and combined display system according to the first embodiment of the present invention when applied to a corresponding input device.

[0024] Figure 2 This is a flowchart of audio and video data synchronization in the first embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is an explanation of the embodiments of the present invention through specific embodiments, and those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the concept of the present invention. In addition, it is stated in advance that the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention. In addition, the term "or" used herein may include any one or more combinations of the associated listed items depending on the actual situation.

[0026] Reference Figure 1 ,in Figure 1 This is a block diagram of the multi-channel medical video acquisition and combined display system of the first embodiment of the present invention when applied to remote interaction during surgery.

[0027] like Figure 1 As shown, the multi-channel medical video acquisition and combined display system based on time synchronization according to the first embodiment of the present invention is applied to remote interaction during surgery and is connected to an input device 1 in an operating room. The system includes a signal acquisition unit 100, a signal processing unit 200, a core processing unit 300, and a display output unit 400. The input device 1 includes: surgical field camera, panoramic camera, C-arm, O-arm, endoscope, laparoscope (not shown), microscope and other instruments in the operating room.

[0028] Specifically, the signal collection unit 100 is configured to collect multiple audio and video signals from input devices.

[0029] Preferably, the interfaces supported by the signal acquisition unit include: HDMI, SDI, DVI, and VGA. Figure 1As shown, for example, it can be configured that the HDMI interface corresponds to the surgical field camera, the SDI interface corresponds to the panoramic camera, the DVI interface corresponds to the C-arm, O-arm and endoscope, and the VGA interface corresponds to the microscope, but it is not limited to this, and the interface can also be adjusted as needed.

[0030] Specific forms of support include:

[0031] HDMI 2.0: supports up to 4K@60Hz resolution;

[0032] 3G-SDI: supports up to 1080p@60Hz resolution;

[0033] DVI-D: supports up to 2560x1600@60Hz resolution;

[0034] VGA: Backward compatible with legacy devices.

[0035] The signal processing unit 200 is configured to convert multiple audio and video signals collected by the signal acquisition unit 100. Preferably, the signal processing unit 200 includes a signal format conversion module and a color space conversion module. Specifically, the signal format conversion module converts the multiple audio and video signals collected by the signal acquisition unit 100 and then processes them by the color space conversion module. The color space conversion module supports automatic conversion of YUV / RGB color space, and can intelligently identify and automatically convert the color space format in the video signal. Whether it is YUV or RGB, it can ensure the true restoration and accurate expression of the image color, and provide flexibility and compatibility for image processing. Moreover, it also realizes the dynamic range mapping from HDR to SDR, which can intelligently convert high dynamic range (HDR) content into standard dynamic range (SDR), while retaining as many image details and color levels as possible, ensuring that high-quality picture effects can be presented on different display devices.

[0036] The core processing unit 300 is configured to generate a display screen after core processing of the multiple audio and video signals converted by the signal processing unit 200. Preferably, the core processing unit 300 includes: a time synchronization processing module for synchronization, a picture combination module for combination, and a timestamp injection module for timestamp injection. Specifically, the synchronization processing module is set as follows on the acquisition channel. Each channel independently collects audio and video data to ensure that each acquisition channel can process the audio and video data stream simultaneously and independently without interfering with each other, thereby improving the efficiency and stability of data acquisition. Each channel has complete processing capabilities and can capture and preliminarily process the input audio and video signals in real time. At the same time, it supports standard input interfaces such as HDMI and SDI. In order to meet the needs of different devices, the acquisition channel supports a variety of mainstream video input interfaces, including high-definition multimedia interface (HDMI) and digital serial interface (SDI). These interfaces are not only highly compatible, but also can transmit high-quality video signals, providing a solid foundation for subsequent audio and video processing. In addition, the collected data is packaged into audio and video data streams, which is convenient for subsequent storage, transmission and processing, ensuring the integrity and consistency of the data, and providing reliable data support for subsequent applications.

[0037] like Figure 2 As shown, the synchronization processing module is also provided with a synchronization buffer. Specifically, a fixed-size buffer queue is configured for each channel. In order to cope with the delay and jitter problems during data transmission, the system configures a fixed-size buffer queue for each acquisition channel. For example, the buffer of each channel can be set to a data volume of 2 seconds to ensure that there is enough buffer space to cope with emergencies during data transmission. At the same time, the buffer queue adopts a FIFO (first-in-first-out) queue structure, that is, the data that enters the buffer first will be processed first. This structure is not only simple, but also can ensure that the processing order of the data is consistent with the input order, thereby avoiding the problem of data confusion. In order to meet the needs of different application scenarios, the size of the buffer can be flexibly configured according to actual conditions. By adjusting the size of the buffer, the delay of data transmission and the resource occupancy of the system can be balanced to achieve the best performance.

[0038] like Figure 2As shown, the synchronization processing module also performs synchronous writing control. Before performing synchronous writing, the system will check whether there is data in the buffers of all acquisition channels. Only when the buffers of all channels are ready for data will the next writing operation be performed, ensuring the synchronization and consistency of multi-channel data. During the synchronous writing process, the system will perform writing operations according to a fixed writing block size (for example, 1 second of data), which not only improves the efficiency of data processing, but also ensures the integrity and continuity of the data. In order to ensure the synchronization and consistency of multi-channel data, the system will maintain the consistency of multi-channel writing positions. During the writing process, the system will ensure that the writing position of each channel is advanced at the same rhythm, thereby avoiding the problem of data misalignment or loss. The time synchronization accuracy is better than 200ms, including 10-30ms for video acquisition, 5-15ms for memory buffer, and 50-150ms for data writing control.

[0039] Preferably, the picture combination module uses GPU acceleration to perform data operations in a zero-copy manner and adaptively adjust the frame rate. Specifically, for example, NVIDIA CUDA technology can be used to achieve GPU acceleration, and a multi-stream parallel processing mechanism is used. In order to maximize the video processing efficiency, an independent CUDA stream is created for each video channel, which not only realizes the parallel processing of multiple videos, but also significantly improves the overall processing speed. At the same time, the CUDA stream synchronization mechanism is used to ensure that the processing tasks of each video channel can be completed synchronously, thereby avoiding the problems of data confusion and processing delay. Use a zero-copy method to reduce memory usage. In order to further optimize memory usage, a direct memory access (DMA) mechanism is introduced. By registering the host memory through CUDA, the GPU can directly access the host memory through the PCIe bus, greatly reducing the explicit memory copy operation from the CPU to the GPU, thereby reducing memory usage and improving processing efficiency. At the same time, a deep optimization of the memory mapping strategy is implemented, and a memory mapping strategy is adopted to directly map the video frame buffer to the GPU address space. The device pointer is obtained through CUDA, and the GPU can directly read and process the video data without additional memory copy steps, further improving the processing speed and system responsiveness. Adaptive adjustment of the frame rate is specifically about dynamic load monitoring and performance optimization. In order to ensure the stable operation of the system under high load, a dynamic load monitoring mechanism is implemented, which can monitor the GPU usage and memory bandwidth in real time, and record the frame processing time and display delay. By setting performance thresholds (such as GPU usage not exceeding 80%), changes in system load can be perceived in a timely manner. Intelligent adjustment of the adaptive strategy is also implemented. When the system load exceeds the preset threshold, the adaptive strategy will be started immediately. By reducing the frame rate of non-main screens, reducing the frequency of screen updates, and appropriately reducing the resolution, the system load is effectively reduced to ensure smooth playback of the main screen. When the load returns to normal, the system will gradually restore the original frame rate and resolution to ensure that users get the best video viewing experience.

[0040] In addition, the screen combination module supports standard layouts such as 1x1, 2x2, 3x3, 1+3, etc., can freely exchange screens at any position, can dynamically adjust the screen size, and can switch screens at any time through shortcut keys and touch operations.

[0041] The timestamp injection module injects timestamps and displays the current time and the relative duration of the surgery in the time information, ensuring that medical staff can grasp the time information of the surgery process in real time. The timestamp supports millisecond-level precision display, providing detailed time data support for surgeries that require high-precision time records. In terms of timestamp display control, position drag adjustment is provided, and users can flexibly adjust the position of the display interface according to actual needs, so as to better perform surgical operations and monitoring.

[0042] The display output unit 400 is configured to display and output the display screen generated by the core processing unit 300 .

[0043] The second embodiment of the present invention is an audio and video interaction method for an operating room, which is a method for audio and video interaction using the multi-channel medical video acquisition and combined display system based on time synchronization of the first embodiment of the present invention. The audio and video interaction method corresponds to the multi-channel medical video acquisition and combined display system of the first embodiment, so the various variations of the first embodiment are also applicable to the second embodiment, which will not be repeated here.

[0044] The following is a specific embodiment to illustrate the multi-channel medical video acquisition and combined display based on time synchronization of the present invention.

[0045] (Example)

[0046] The time-synchronized multi-channel medical video acquisition and combined display system of the present invention is applied to multi-device collaborative surgery (such as neurosurgery).

[0047] For three video signals, the surgical microscope is configured to correspond to HDMI output, the surgical field camera is configured to correspond to SDI output), and the navigation system is configured to correspond to DVI output. The signal acquisition unit receives three video signals at the same time, and the time synchronization processing module ensures that the three signals are time-aligned. The timestamp injection module injects the surgical timing information as a timestamp, the picture combination module uses a 1+2 layout display, and the real-time display module outputs the display picture to the large screen in the operating room.

[0048] In this way, the three-way images are ensured to be fully synchronized, with a display delay of <20ms and clear and stable image quality.

[0049] As described above, the multi-channel medical video acquisition and combined display technology based on time synchronization provided by the present invention solves the problem of unified management of multi-device video acquisition, supports multiple video interfaces and formats, and provides device plug-and-play capabilities. Accurate time synchronization of multi-channel videos is achieved to ensure multi-channel video frame alignment. The display effect is significantly improved, the end-to-end delay is <20ms, and 4K ultra-high-definition display is supported. Flexible layout adjustment capabilities are provided, time information management is enhanced, time reference is unified, display mode is flexible, and expansion functions are rich.

[0050] Industrial Applicability

[0051] In summary, the present invention provides a multi-channel medical video acquisition and combined display system and method based on time synchronization. Compared with the prior art, it solves the problem of unified management of multi-device video acquisition, realizes accurate time synchronization of multi-channel videos, significantly improves display effects, and enhances time information management.

[0052] The contents disclosed above are only preferred feasible embodiments of the present invention, and are not intended to limit the scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention description and drawings are included in the scope of the present invention.

Claims

1. A multi-channel medical video acquisition and combined display system based on time synchronization, characterized in that: Include: A signal acquisition unit configured to acquire a plurality of audio and video signals from an input device; A signal processing unit configured to convert a plurality of audio and video signals collected by the signal collection unit; The core processing unit is configured to generate a display screen after performing core processing on the multiple audio and video signals converted by the signal processing unit, and the core processing includes: synchronization, combination, and time stamp injection; The display output unit is configured to display and output the display screen generated by the core processing unit.

2. The multi-channel medical video acquisition and combined display system according to claim 1, characterized in that: The signal acquisition unit supports interfaces including: HDMI, SDI, DVI, and VGA.

3. The multi-channel medical video acquisition and combined display system according to claim 1, characterized in that: The signal processing unit includes a signal format conversion module and a color space conversion module.

4. The multi-channel medical video acquisition and combined display system according to claim 1, characterized in that: The core processing unit includes: a time synchronization processing module for synchronization, a picture combination module for combination, and a timestamp injection module for timestamp injection.

5. The multi-channel medical video acquisition and combined display system as claimed in claim 4, characterized in that: The synchronization processing module performs synchronization by setting a buffer queue.

6. The multi-channel medical video acquisition and combined display system according to claim 4, characterized in that: The picture combination module uses GPU acceleration to perform data operations in a zero-copy manner and adaptively adjusts the frame rate.

7. The multi-channel medical video acquisition and combined display system according to claim 6, characterized in that: The picture combination module performs 2x2 or 3x3 picture combination.

8. The multi-channel medical video acquisition and combined display system according to claim 1, characterized in that: The display output unit also includes a storage and recording module.

9. An audio and video interaction method, characterized in that: The invention relates to a method for performing audio and video interaction using the multi-channel medical video acquisition and combined display system as described in any one of claims 1 to 8.

Images