Remote operation video optimization system based on eye movement tracking

Abstract

The invention provides a remote operation video optimization system based on eye movement tracking, which combines eye movement tracking data and a visual attention model to dynamically generate a region of interest in an operation video; respectively carrying out differential compression on the region of interest and the non-region of interest, preferentially allocating bandwidth for the region of interest, coding the region of interest by adopting high resolution so as to ensure that key details in the region of interest are clear, and simultaneously reducing bandwidth occupation of the non-region of interest; a video bandwidth allocation strategy is adjusted in real time based on the network state, so that network resource allocation is adjusted, and the transmission quality of the region of interest is guaranteed preferentially; therefore, the delay can be remarkably reduced through dynamic interested generation and coding optimization, and the real-time performance and reliability of the remote operation are improved; meanwhile, by optimizing the display quality of the key area, the visual search burden of the doctor can be reduced, and the operation efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical robots and remote surgery, and in particular relates to a remote surgery video optimization system based on eye tracking. Background Art

[0002] Remote surgery is a technology that transmits surgeons’ operating instructions to remote robotic arms in real time over the network to complete the operation. Real-time transmission of surgical videos is a key link in remote surgery systems. However, due to limited network bandwidth, the transmission of high-definition surgical videos is susceptible to freezes and delays, which reduces the safety and accuracy of the surgery.

[0003] Existing methods mainly use video compression algorithms and fixed bandwidth allocation strategies, but fail to fully combine the surgeon's visual focus to dynamically adjust video encoding parameters, resulting in inefficient use of precious bandwidth resources. Eye tracking technology can accurately capture the surgeon's visual attention, making it possible to dynamically optimize surgical videos.

[0004] That is to say, the existing video compression algorithm and fixed bandwidth allocation strategy have the following defects:

[0005] 1. Insufficient resource utilization: Traditional methods use a unified encoding strategy for all areas of surgical videos, failing to distinguish between key and secondary areas, resulting in a waste of bandwidth resources.

[0006] 2. High transmission delay: The fixed bandwidth allocation strategy cannot adapt to fluctuations in network conditions and can easily cause video freezes when bandwidth is insufficient, affecting the real-time and reliability of remote surgery.

[0007] 3. Heavy operating burden on doctors: Existing systems do not optimize video quality based on the surgeon’s visual focus area. Doctors need to look for surgical details in blurry non-critical areas, which increases the difficulty of operation.

[0008] 4. Lack of dynamism: Existing methods lack the ability to dynamically adjust video encoding and bandwidth allocation based on real-time network status and doctor needs. Summary of the invention

[0009] To solve the above problems, the present invention provides a remote surgery video optimization system based on eye tracking, which obtains the surgeon's visual focus area in real time, dynamically adjusts the surgery video encoding parameters, optimizes bandwidth allocation, and achieves efficient transmission and precise surgery.

[0010] A remote surgery video optimization system based on eye tracking, comprising an eye movement data acquisition module, a region of interest generation module, a video encoding optimization module, a dynamic bandwidth allocation module, and a video decoding and presentation module;

[0011] The eye movement data acquisition module is used to capture the doctor's gaze point in real time and generate sight track data;

[0012] The region of interest generation module is used to cluster the sight trajectory data to generate a region of interest;

[0013] The video coding optimization module is used to encode the region of interest and the region of no interest respectively, wherein the coding resolution of the region of interest is higher than the coding resolution of the region of no interest;

[0014] The dynamic bandwidth allocation module is used to adjust the bandwidth allocation ratio between the region of interest and the region of no interest in real time, wherein the bandwidth of the region of interest is greater than the bandwidth of the region of no interest;

[0015] The video decoding and presentation module is used to present the video stream encoded according to the set encoding resolution and transmitted according to the set bandwidth to the surgeon at the remote surgery terminal.

[0016] Furthermore, the eye movement data acquisition module captures the doctor's gaze point in real time through an eye movement tracking device installed in the surgical terminal.

[0017] Furthermore, the eye tracking device is a binocular infrared eye tracker, a physiological sensor or a module built into AR glasses.

[0018] Furthermore, the region of interest generation module clusters the sight trajectory data based on a visual attention network model to generate a region of interest.

[0019] Furthermore, the dynamic bandwidth allocation module uses a dynamic bandwidth allocation algorithm based on reinforcement learning to adjust the bandwidth allocation ratio between the region of interest and the region of no interest in real time.

[0020] Furthermore, the dynamic bandwidth allocation module uses the congestion status of the remote transmission network and the size and location of the region of interest as inputs to a dynamic bandwidth allocation algorithm based on reinforcement learning, and the dynamic bandwidth allocation algorithm outputs the bandwidth allocation ratio of the region of interest to the region of non-interest in real time.

[0021] Beneficial effects:

[0022] The present invention provides a remote surgery video optimization system based on eye tracking, which combines eye tracking data with a visual attention model to dynamically generate regions of interest in surgical videos; differential compression is performed on regions of interest and non-regions of interest, bandwidth is allocated preferentially to regions of interest, and high resolution is used to encode regions of interest to ensure that key details in regions of interest are clear while reducing bandwidth occupancy in regions of non-interest; the present invention also adjusts the video bandwidth allocation strategy in real time based on network status to adjust network resource allocation and prioritize transmission quality in regions of interest; therefore, the present invention can significantly reduce latency through dynamic interest generation and coding optimization, thereby improving the real-time and reliability of remote surgery; at the same time, the present invention can also reduce the visual search burden of doctors and improve operating efficiency by optimizing the display quality of key areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A principle block diagram of a remote surgery video optimization system based on eye tracking provided by the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0025] The present invention aims to provide a system for generating a region of interest (ROI) in a remote surgery video based on eye tracking technology, and to improve the video transmission efficiency and the accuracy of surgical operations by dynamically optimizing bandwidth allocation.

[0026] Specifically, Figure 1 As shown, a remote surgery video optimization system based on eye tracking includes an eye movement data acquisition module, a region of interest generation module, a video encoding optimization module, a dynamic bandwidth allocation module, and a video decoding and presentation module.

[0027] The eye movement data acquisition module is used to capture the doctor's gaze point in real time and generate eye trajectory data; wherein, the eye movement data acquisition module captures the doctor's gaze point in real time through an eye tracking device installed on the surgical terminal; the eye tracking device can use different technical routes, such as binocular infrared eye trackers, physiological sensors or AR glasses built-in modules.

[0028] The region of interest generation module is used to cluster the sight track data based on the visual attention network model to generate the region of interest. For example, the present invention can use an open source eye tracking data processing library to perform sight track analysis, thereby dividing the region of interest and the region of non-interest in the surgical video.

[0029] The video coding optimization module is used to encode the region of interest and the region of no interest respectively, wherein the coding resolution of the region of interest is higher than the coding resolution of the region of no interest.

[0030] The dynamic bandwidth allocation module adopts a dynamic bandwidth allocation algorithm based on reinforcement learning to adjust the bandwidth allocation ratio between the area of ​​interest and the area of ​​non-interest in real time, wherein the bandwidth of the area of ​​interest is greater than the bandwidth of the area of ​​non-interest; wherein the dynamic bandwidth allocation module uses the congestion state of the remote transmission network and the size and position of the area of ​​interest as inputs of the dynamic bandwidth allocation algorithm based on reinforcement learning, and the dynamic bandwidth allocation algorithm outputs the bandwidth allocation ratio between the area of ​​interest and the area of ​​non-interest in real time.

[0031] It should be noted that other technical routes can be chosen for the region of interest ROI generation and bandwidth allocation optimization algorithm, such as using a deep learning model to predict the doctor's visual interest points, or adopting a distributed network control strategy to optimize bandwidth allocation.

[0032] The video decoding and presentation module is used to present the video stream encoded according to the set encoding resolution and transmitted according to the set bandwidth to the surgeon at the remote surgery terminal.

[0033] It should be noted that the hardware configuration required by the remote surgery video optimization system of the present invention includes: installing an eye tracking device (such as a binocular infrared eye tracker) at the surgical terminal, cooperating with a high-definition video acquisition device and a network transmission module. Dedicated computing nodes are configured for real-time processing of eye tracking data and video frames.

[0034] Furthermore, the present invention provides a remote surgery video optimization system based on eye tracking, and the working process is as follows:

[0035] Step 1: Eye tracking data collection

[0036] The eye tracking device installed at the surgical terminal captures the doctor's visual focus in real time and generates gaze trajectory data.

[0037] Step 2: Generate Region of Interest

[0038] The regions of interest are dynamically calibrated in the surgical video frames according to the gaze trajectories, and the gaze trajectories are clustered using a method based on the visual attention model, so as to dynamically generate accurate regions of interest by combining eye tracking data with surgical videos.

[0039] Step 3: Video encoding optimization

[0040] Different encoding compression ratios are used for regions of interest and non-regions of interest, and bandwidth is allocated to regions of interest first. That is, high-resolution encoding is used for regions of interest to ensure the clarity of key surgical details; low-resolution encoding is used for regions of non-interest to save bandwidth.

[0041] Step 4: Dynamic Bandwidth Allocation

[0042] Based on the network status and video content characteristics, the bandwidth allocation ratio of the interested and non-interested areas is adjusted in real time to give priority to ensuring the video transmission quality in the interested areas, thereby optimizing the video transmission quality under limited network conditions through dynamic bandwidth allocation strategies.

[0043] Step 5: Video decoding and presentation

[0044] At the remote surgery terminal, the optimized video stream is decoded and presented to the surgeon, ensuring the real-time and accuracy of the operation.

[0045] In summary, the present invention has the following advantages over the existing methods:

[0046] 1. Efficient bandwidth utilization: Compared with traditional methods, the present invention uses eye tracking technology to achieve accurate allocation of bandwidth resources and reduce bandwidth waste in non-critical areas.

[0047] 2. Enhanced real-time performance: The existing technology has failed to effectively solve the problem of video transmission delay, while this application significantly reduces the delay through dynamic ROI generation and encoding optimization, thereby improving the real-time performance and reliability of remote surgery.

[0048] 3. Improved operational convenience: By optimizing the display quality of key areas, the visual search burden of doctors is reduced and operational efficiency is improved.

[0049] 4. Strong network adaptability: Dynamically adjust encoding and bandwidth allocation based on network status, significantly improving the system's adaptability in low-bandwidth environments.

[0050] 5. Wide application scenarios: The present invention can be extended to other remote collaboration scenarios, such as remote teaching or industrial control.

[0051] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A remote surgery video optimization system based on eye tracking, characterized in that: It includes an eye movement data acquisition module, an area of ​​interest generation module, a video encoding optimization module, a dynamic bandwidth allocation module, and a video decoding and presentation module; The eye movement data acquisition module is used to capture the doctor's gaze point in real time and generate sight track data; The region of interest generation module is used to cluster the sight trajectory data to generate a region of interest; The video coding optimization module is used to encode the region of interest and the region of no interest respectively, wherein the coding resolution of the region of interest is higher than the coding resolution of the region of no interest; The dynamic bandwidth allocation module is used to adjust the bandwidth allocation ratio between the region of interest and the region of no interest in real time, wherein the bandwidth of the region of interest is greater than the bandwidth of the region of no interest; The video decoding and presentation module is used to present the video stream encoded according to the set encoding resolution and transmitted according to the set bandwidth to the surgeon at the remote surgery terminal.

2. The remote surgery video optimization system based on eye tracking according to claim 1, characterized in that: The eye movement data acquisition module captures the doctor's gaze point in real time through an eye movement tracking device installed in the surgical terminal.

3. The remote surgery video optimization system based on eye tracking as claimed in claim 2, characterized in that: The eye tracking device is a binocular infrared eye tracker, a physiological sensor or a built-in module of AR glasses.

4. The remote surgery video optimization system based on eye tracking according to claim 1, characterized in that: The region of interest generation module clusters the sight trajectory data based on the visual attention network model to generate the region of interest.

5. The remote surgery video optimization system based on eye tracking according to claim 1, characterized in that: The dynamic bandwidth allocation module uses a dynamic bandwidth allocation algorithm based on reinforcement learning to adjust the bandwidth allocation ratio between the region of interest and the region of no interest in real time.

6. The remote surgery video optimization system based on eye tracking as claimed in claim 5, characterized in that: The dynamic bandwidth allocation module uses the congestion status of the remote transmission network and the size and location of the area of ​​interest as inputs to the dynamic bandwidth allocation algorithm based on reinforcement learning, and the dynamic bandwidth allocation algorithm outputs the bandwidth allocation ratio of the area of ​​interest to the area of ​​non-interest in real time.

Images