A method, system and storage medium for visualization of AI laparoscope based on focus tracking

Abstract

The present invention discloses a method, system and storage medium for visualization of an AI laparoscope based on focus tracking, which relates to the field of medical information recognition and tracking. An AI laparoscope is used to adjust different angles and brightness to generate a full-wide-angle and omnidirectional video of the region of interest; key frame images are extracted from the video of the region of interest, and the key frame images include images of the lesion site and images containing both the AI ​​laparoscope body and surgical instruments; a deep learning algorithm is used to simultaneously extract image features of different scales corresponding to the key frame images, and a lens tracking signal and an obstacle avoidance control signal are generated; the lens tracking signal has a higher priority than the obstacle avoidance control signal to achieve efficient tracking of the lesion site, and the video of the region of interest is reconstructed based on the tracking of the lesion site by the lens tracking signal and the avoidance path of the AI ​​laparoscope to complete the AI ​​laparoscope visualization. The present invention uses AI technology to achieve focus tracking, video splicing, feature extraction and signal generation to make laparoscope visualization more intelligent.

Description

Technical Field

[0001] The present invention relates to the field of medical information recognition and tracking, and more specifically to a method, system and storage medium for AI laparoscope visualization based on focus tracking. Background Art

[0002] With the advancement of medical technology, surgical procedures are becoming increasingly demanding in precision. This is especially true in complex minimally invasive procedures, such as endoscopic neurosurgery and laparoscopic urology, where errors as small as a millimeter can lead to serious complications. The introduction of focus tracking technology can help doctors better observe the details of the surgical site and improve surgical precision. For example, in ophthalmic vitrectomy, accurate tracking of key areas such as the retina can reduce damage to surrounding tissues.

[0003] In recent years, artificial intelligence has made tremendous progress in medical image analysis. In diagnostic imaging such as X-rays, CT scans, and MRIs, AI algorithms are already helping doctors detect lesions more quickly and accurately. The introduction of AI technology for focus tracking in laparoscopy is an extension of this trend. AI can analyze laparoscopic images in real time and, by learning from vast amounts of surgical imaging data, automatically identify key anatomical structures and lesion areas during surgery.

[0004] For example, using convolutional neural networks (CNNs), a deep learning algorithm, it is possible to extract and classify different tissue features in laparoscopic images. When a doctor manipulates instruments close to critical tissue during surgery, the AI ​​system can promptly adjust the focus to ensure that key areas are clearly visible, acting as an intelligent "visual assistant" for the doctor.

[0005] While some laparoscopic-assisted devices, such as robotic-assisted laparoscopic operating systems, are currently available, these systems primarily focus on controlling the position and angle of the laparoscope, and automated focus tracking is still inadequate. Furthermore, some existing focus adjustment methods may require the surgeon to manually adjust the focus using knobs, which can distract the surgeon and prolong the procedure. Furthermore, manual adjustment cannot meet the speed and accuracy requirements of complex surgeries. Summary of the Invention

[0006] In view of this, the present invention provides a method, system and storage medium for AI laparoscope visualization based on focus tracking, which uses AI technology to realize a series of operations such as focus tracking, video stitching, feature extraction and signal generation, so that laparoscope visualization has a higher level of intelligence.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A focus tracking AI laparoscope visualization method includes the following steps:

[0009] Use AI laparoscope to adjust different angles and brightness, collect different wide-angle videos and non-wide-angle videos, and complete stitching to generate full-wide-angle and all-round videos of the area of ​​interest;

[0010] Extract key frame images from the video of the region of interest. The key frame images include images of the lesion site and images containing both the AI ​​laparoscope body and surgical instruments.

[0011] Using deep learning algorithms to simultaneously extract image features of different scales corresponding to keyframe images, the camera tracking signal and obstacle avoidance control signal are generated.

[0012] The lens tracking signal has a higher priority than the obstacle avoidance control signal to achieve efficient tracking of the lesion site. The AI ​​laparoscope visualization is completed by tracking the lesion site based on the lens tracking signal and reconstructing the video of the area of ​​interest based on the AI ​​laparoscope avoidance path.

[0013] Optionally, based on the position of surgical instruments, the tissue type of the area of ​​interest and preset lighting standards, the brightness, color and light distribution pattern of the LED light source are dynamically adjusted to achieve efficient fill light for the lens.

[0014] Optionally, the specific steps for generating a full wide-angle and omnidirectional video of the region of interest are as follows:

[0015] The wide-angle video and non-wide-angle video corresponding frame pictures are extracted respectively. One wide-angle frame picture corresponds to multiple non-wide-angle frame pictures. Multiple A frames are set on the wide-angle frame picture. i Feature points, different non-wide-angle frame pictures correspond to the a i Feature points;

[0016] A i Feature points and a i The feature points are used to form a wide-angle frame matrix and a non-wide-angle frame matrix, respectively. The similarity between the wide-angle frame matrix and the non-wide-angle frame matrix is ​​calculated, and matching is performed on the wide-angle frame matrix and the non-wide-angle frame matrix based on the similarity result.

[0017] Based on the matching results, the wide-angle frame image and non-wide-angle frame image corresponding to the feature points are determined, and the wide-angle frame image and non-wide-angle frame image are spliced. The edge algorithm is used to de-marginalize the spliced ​​edges to obtain a full wide-angle and full-dimensional image of the region of interest, and a full wide-angle and full-dimensional video of the region of interest is generated according to the corresponding frame.

[0018] Optionally, the specific working process of the obstacle avoidance control signal is: when it is detected that the distance between the AI ​​laparoscope body and the surgical instrument is less than the preset safety threshold, the obstacle avoidance algorithm is triggered. The obstacle avoidance algorithm plans a safe AI laparoscope avoidance path based on the pre-built surgical space model and real-time sensor data.

[0019] Optionally, AI laparoscope avoidance path planning is as follows:

[0020] Simplify the AI ​​laparoscope and surgical instruments, retain the peripheral geometric features and size parameters, use the AI ​​laparoscope as a fixed point, calculate the distance of the surgical instruments, and set the preset safe distance;

[0021] The motion trajectory of the surgical instrument is predicted based on the relative position of the lens tracking signal and the area of ​​interest, and the avoidance path planning of the AI ​​laparoscope is determined based on the motion trajectory and safety distance.

[0022] Optionally, a deep learning algorithm performs top-down forward propagation on key frame images based on a feature pyramid network architecture to extract image features of different scales.

[0023] A focus tracking AI laparoscope visualization system, comprising:

[0024] Region of Interest Video Stitching Module: Used to use AI laparoscope to adjust different angles and brightness, collect different wide-angle videos and non-wide-angle videos, and complete stitching to generate full-wide-angle and full-dimensional region of interest videos;

[0025] Key frame image extraction module: used to extract key frame images from the video of the area of ​​interest. Key frame images include images of the lesion site and images containing both the AI ​​laparoscope body and surgical instruments.

[0026] Lens tracking signal and obstacle avoidance control signal generation module: used to simultaneously extract image features of different scales corresponding to key frame images using a deep learning algorithm to generate lens tracking signals and obstacle avoidance control signals;

[0027] AI laparoscope visualization module: The lens tracking signal has a higher priority than the obstacle avoidance control signal to achieve efficient tracking of the lesion site. The AI ​​laparoscope visualization is completed by tracking the lesion site based on the lens tracking signal and reconstructing the video of the area of ​​interest based on the AI ​​laparoscope avoidance path.

[0028] Optionally, it also includes a lens fill light module: used to dynamically adjust the brightness, color and light distribution pattern of the LED light source based on the position of surgical instruments, tissue type of the area of ​​interest and preset lighting standards, so as to achieve efficient fill light for the lens.

[0029] A computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of any one of the aforementioned methods for visualization of an AI laparoscope based on focus tracking are implemented.

[0030] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a method, system, and storage medium for visualization of an AI laparoscope based on focus tracking, which have the following beneficial effects:

[0031] 1. The AI ​​laparoscope automatically adjusts its angle and brightness to accurately capture lesions during surgery, especially in complex or difficult-to-observe areas. This precision helps doctors more accurately diagnose conditions and improve surgical success rates.

[0032] 2. By stitching together wide-angle and non-wide-angle videos, a full-width, all-around video of the region of interest can be generated, providing doctors with a wider surgical field of view. This helps doctors better understand the overall situation of the surgery and reduce operational errors caused by a limited field of view.

[0033] 3. By using a deep learning algorithm to extract keyframe images and their corresponding image features, and generating lens tracking and obstacle avoidance control signals, the AI ​​laparoscope can efficiently track the lesion while avoiding collisions with surgical instruments. This significantly reduces the time and effort required by the surgeon to manually adjust the laparoscope during surgery, improving surgical efficiency.

[0034] 4. Because the AI ​​laparoscope can automatically track the lesion and avoid surgical instruments, it can reduce surgical risks caused by improper operation or misjudgment. At the same time, the reconstruction of full-width video also helps doctors better identify potential risk factors and take timely and effective countermeasures. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0036] Figure 1 Schematic diagram of the method flow of the present invention;

[0037] Figure 2 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] The embodiment of the present invention discloses a method for visualizing an AI laparoscope based on focus tracking, such as Figure 1 As shown, the following steps are included:

[0040] Step 1: Use the AI ​​laparoscope to adjust different angles and brightness, collect different wide-angle videos and non-wide-angle videos, and complete the stitching to generate a full-wide-angle and all-around video of the area of ​​interest;

[0041] Step 2: Extract key frame images from the video of the region of interest. The key frame images include images of the lesion site and images of both the AI ​​laparoscope body and surgical instruments.

[0042] Step 3: Use a deep learning algorithm to simultaneously extract image features of different scales corresponding to the key frame images to generate lens tracking signals and obstacle avoidance control signals;

[0043] Step 4: The lens tracking signal has a higher priority than the obstacle avoidance control signal to achieve efficient tracking of the lesion. Based on the tracking of the lesion by the lens tracking signal and the video reconstruction of the area of ​​interest based on the AI ​​laparoscope avoidance path, the AI ​​laparoscope visualization is completed.

[0044] Furthermore, this embodiment also includes dynamically adjusting the brightness, color and light distribution pattern of the LED light source based on the position of the surgical instrument, the tissue type of the area of ​​interest and the preset lighting standard to achieve efficient fill light for the lens.

[0045] Furthermore, in step 1, the specific steps for generating a full wide-angle and omnidirectional video of the region of interest are as follows:

[0046] Step 1.1, respectively extract the wide-angle video and non-wide-angle video corresponding frame pictures, one wide-angle frame picture corresponds to multiple non-wide-angle frame pictures, and multiple A-numbers are set on the wide-angle frame picture. i Feature points, different non-wide-angle frame pictures correspond to the a i Feature points;

[0047] Step 1.2, the A i Feature points and a i The feature points are used to form a wide-angle frame matrix and a non-wide-angle frame matrix, respectively. The similarity between the wide-angle frame matrix and the non-wide-angle frame matrix is ​​calculated, and matching is performed on the wide-angle frame matrix and the non-wide-angle frame matrix based on the similarity result.

[0048] Step 1.3: Based on the matching results, determine the wide-angle frame image and non-wide-angle frame image corresponding to the feature point, and splice the wide-angle frame image and non-wide-angle frame image. Use the edge algorithm to de-marginalize the spliced ​​edges to obtain a full wide-angle and full-dimensional image of the region of interest, and generate a full wide-angle and full-dimensional video of the region of interest according to the corresponding frame.

[0049] Furthermore, a reasonable similarity threshold is set. This threshold can be determined through experimental analysis, statistics on samples with known good and poor matches, etc. Then, the calculated similarity values ​​between the wide-angle frame matrix and the non-wide-angle frame matrix (whether calculated element by element or as a whole) are compared with this threshold. When the similarity value of a pair of wide-angle frame images and non-wide-angle frame images (or the entire matrix set) is greater than or equal to the threshold, they are considered to be matched, that is, the two images (or two groups) are deemed to have sufficient similarity in content and can be operated as corresponding image pairs in subsequent processing, such as image fusion, feature alignment, and other related processing.

[0050] Furthermore, in step three, the specific working process of the obstacle avoidance control signal is: when it is detected that the distance between the AI ​​laparoscope body and the surgical instrument is less than the preset safety threshold, the obstacle avoidance algorithm is triggered. The obstacle avoidance algorithm plans a safe AI laparoscope avoidance path based on the pre-built surgical space model and real-time sensor data.

[0051] Furthermore, in step 4, the AI ​​laparoscope avoidance path planning is specifically as follows:

[0052] Step 4.1: Simplify the AI ​​laparoscope and surgical instruments, retaining their peripheral geometric features and dimensional parameters. Using the AI ​​laparoscope as a fixed point, calculate the distance between the surgical instruments and the preset safety distance. Statistically analyze the distance values ​​of all sampled points, such as taking the minimum or average value, or determining a reasonable distance threshold based on a certain safety factor. This threshold is set as the preset safety distance. Determining this safety distance requires comprehensive consideration of factors such as the operational flexibility of the surgical instruments, the precision requirements of the surgery, and the fragile areas and sensitive components of the AI ​​laparoscope. This ensures that collisions between the surgical instruments and the AI ​​laparoscope can be effectively avoided, even with certain operational errors and uncertainties during actual surgery.

[0053] Step 4.2: Predict the trajectory of the surgical instrument based on the relative position of the lens tracking signal and the region of interest, and determine the avoidance path planning of the AI ​​laparoscope based on the trajectory and safety distance.

[0054] Specifically, by analyzing the time series of camera tracking signals and their relative positional changes with the region of interest, machine learning models or prediction methods based on physical kinematics are used to predict the trajectory of surgical instruments over a period of time. These prediction methods can take into account factors such as the surgeon's operating habits, the velocity and acceleration patterns of surgical instruments, and the constraints of the surgical scenario to improve the accuracy and reliability of trajectory prediction.

[0055] Based on the predicted trajectory of the surgical instrument and the previously calculated safety distance, the AI ​​laparoscope is used as the operating object to perform path planning calculations in the virtual surgical environment. First, the trajectory of the surgical instrument is discretized, breaking it down into a series of time steps and spatial locations. Then, for each time step, the safe position area that the AI ​​laparoscope needs to avoid for the surgical instrument is calculated.

[0056] Using a path search algorithm, such as the A* algorithm, Dijkstra algorithm, or the sampling-based Rapid Exploration Random Tree (RRT) algorithm, a collision-free avoidance path is searched within the feasible motion space of the AI ​​laparoscope from the current position to the target position (e.g., the optimal observation position in the surgical field of view or the next observation point pre-set according to the surgical process), while meeting safety distance constraints. This path should be as smooth and continuous as possible to reduce vibration and instability during the movement of the AI ​​laparoscope. The length and movement time of the path should also be considered to prevent excessive avoidance movements from affecting the efficiency and smoothness of the surgery.

[0057] After the avoidance path is determined, the path information is converted into actual control instructions and sent to the drive device or motion control system of the AI ​​laparoscope, so that it can automatically perform avoidance movements according to the planned path, or provide the surgeon with visual avoidance prompts and operation suggestions so that the laparoscope position can be manually adjusted when necessary to ensure the safety and smooth progress of the operation.

[0058] Furthermore, the deep learning algorithm performs top-down forward propagation on the key frame images based on the feature pyramid network architecture to extract image features of different scales.

[0059] and Figure 1 Corresponding to the method shown, the present invention also discloses a focus tracking AI laparoscope visualization system for Figure 1 The implementation of the method, the specific structure is as follows Figure 2 Shown, including:

[0060] Region of Interest Video Stitching Module: Used to use AI laparoscope to adjust different angles and brightness, collect different wide-angle videos and non-wide-angle videos, and complete stitching to generate full-wide-angle and full-dimensional region of interest videos;

[0061] Key frame image extraction module: used to extract key frame images from the video of the area of ​​interest. Key frame images include images of the lesion site and images containing both the AI ​​laparoscope body and surgical instruments.

[0062] Lens tracking signal and obstacle avoidance control signal generation module: used to simultaneously extract image features of different scales corresponding to key frame images using a deep learning algorithm to generate lens tracking signals and obstacle avoidance control signals;

[0063] AI laparoscope visualization module: The lens tracking signal has a higher priority than the obstacle avoidance control signal to achieve efficient tracking of the lesion site. The AI ​​laparoscope visualization is completed by tracking the lesion site based on the lens tracking signal and reconstructing the video of the area of ​​interest based on the AI ​​laparoscope avoidance path.

[0064] Optionally, it also includes a lens fill light module: used to dynamically adjust the brightness, color and light distribution pattern of the LED light source based on the position of surgical instruments, tissue type of the area of ​​interest and preset lighting standards, so as to achieve efficient fill light for the lens.

[0065] Finally, this embodiment also discloses a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, any one of the steps of the focus tracking-based AI laparoscope visualization method is implemented.

[0066] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0067] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for visualization of AI laparoscope based on focus tracking, characterized in that: The following steps are involved: Use AI laparoscope to adjust different angles and brightness, collect different wide-angle videos and non-wide-angle videos, and complete stitching to generate full-wide-angle and all-round videos of the area of ​​interest; The specific steps to generate a full wide-angle and omnidirectional video of the region of interest are as follows: The wide-angle video and non-wide-angle video corresponding frame pictures are extracted respectively. One wide-angle frame picture corresponds to multiple non-wide-angle frame pictures. Multiple A frames are set on the wide-angle frame picture. i Feature points, different non-wide-angle frame pictures correspond to the a i Feature points; A i Feature points and a i The feature points are used to form a wide-angle frame matrix and a non-wide-angle frame matrix, respectively. The similarity between the wide-angle frame matrix and the non-wide-angle frame matrix is ​​calculated, and matching is performed on the wide-angle frame matrix and the non-wide-angle frame matrix based on the similarity result. Based on the matching results, the wide-angle frame image and non-wide-angle frame image corresponding to the feature point are determined, and the wide-angle frame image and non-wide-angle frame image are spliced ​​together. The edge algorithm is used to de-marginalize the spliced ​​edges to obtain a full wide-angle and full-range image of the region of interest, and a full wide-angle and full-range video of the region of interest is generated according to the corresponding frame; Extract key frame images from the video of the region of interest. The key frame images include images of the lesion site and images containing both the AI ​​laparoscope body and surgical instruments. Using deep learning algorithms to simultaneously extract image features of different scales corresponding to keyframe images, the camera tracking signal and obstacle avoidance control signal are generated. The lens tracking signal takes priority over the obstacle avoidance control signal to achieve efficient tracking of the lesion. Based on the tracking of the lesion by the lens tracking signal and the video reconstruction of the region of interest based on the AI ​​laparoscope's avoidance path, AI laparoscope visualization is completed. The AI ​​laparoscope avoidance path planning is as follows: Simplify the AI ​​laparoscope and surgical instruments, retain the peripheral geometric features and size parameters, use the AI ​​laparoscope as a fixed point, calculate the distance of the surgical instruments, and set the preset safe distance; The motion trajectory of the surgical instrument is predicted based on the relative position of the lens tracking signal and the area of ​​interest, and the avoidance path planning of the AI ​​laparoscope is determined based on the motion trajectory and safety distance.

2. The method for visualization of an AI laparoscope based on focus tracking according to claim 1, characterized in that: Based on the position of surgical instruments, the tissue type of the area of ​​interest and the preset lighting standards, the brightness, color and light distribution pattern of the LED light source are dynamically adjusted to achieve efficient fill light for the lens.

3. The method for visualization of an AI laparoscope based on focus tracking according to claim 1, characterized in that: The specific working process of the obstacle avoidance control signal is: when it is detected that the distance between the AI ​​laparoscope body and the surgical instruments is less than the preset safety threshold, the obstacle avoidance algorithm is triggered. The obstacle avoidance algorithm plans a safe AI laparoscope avoidance path based on the pre-built surgical space model and real-time sensor data.

4. The method for visualization of an AI laparoscope based on focus tracking according to claim 1, characterized in that: The deep learning algorithm performs top-down forward propagation on key frame images based on the feature pyramid network architecture to extract image features of different scales.

5. A focus tracking AI laparoscope visualization system, characterized in that: include: Region of Interest Video Stitching Module: Used to use AI laparoscope to adjust different angles and brightness, collect different wide-angle videos and non-wide-angle videos, and complete stitching to generate full-wide-angle and full-dimensional region of interest videos; The specific steps to generate a full wide-angle and omnidirectional video of the region of interest are as follows: The wide-angle video and non-wide-angle video corresponding frame pictures are extracted respectively. One wide-angle frame picture corresponds to multiple non-wide-angle frame pictures. Multiple A frames are set on the wide-angle frame picture. i Feature points, different non-wide-angle frame pictures correspond to the a i Feature points; A i Feature points and a i The feature points are used to form a wide-angle frame matrix and a non-wide-angle frame matrix, respectively. The similarity between the wide-angle frame matrix and the non-wide-angle frame matrix is ​​calculated, and matching is performed on the wide-angle frame matrix and the non-wide-angle frame matrix based on the similarity result. Based on the matching results, the wide-angle frame image and non-wide-angle frame image corresponding to the feature point are determined, and the wide-angle frame image and non-wide-angle frame image are spliced ​​together. The edge algorithm is used to de-marginalize the spliced ​​edges to obtain a full wide-angle and full-range image of the region of interest, and a full wide-angle and full-range video of the region of interest is generated according to the corresponding frame; Key frame image extraction module: used to extract key frame images from the video of the area of ​​interest. Key frame images include images of the lesion site and images containing both the AI ​​laparoscope body and surgical instruments. Lens tracking signal and obstacle avoidance control signal generation module: used to simultaneously extract image features of different scales corresponding to key frame images using a deep learning algorithm to generate lens tracking signals and obstacle avoidance control signals; AI laparoscope visualization module: This module prioritizes lens tracking signals over obstacle avoidance control signals to achieve efficient tracking of lesions. This module reconstructs the video of the region of interest based on the lens tracking signal and the AI ​​laparoscope avoidance path, completing AI laparoscope visualization. The AI ​​laparoscope avoidance path planning is as follows: Simplify the AI ​​laparoscope and surgical instruments, retain the peripheral geometric features and size parameters, use the AI ​​laparoscope as a fixed point, calculate the distance of the surgical instruments, and set the preset safe distance; The motion trajectory of the surgical instrument is predicted based on the relative position of the lens tracking signal and the area of ​​interest, and the avoidance path planning of the AI ​​laparoscope is determined based on the motion trajectory and safety distance.

6. The focus tracking AI laparoscope visualization system according to claim 5, characterized in that: It also includes a lens fill light module: it is used to dynamically adjust the brightness, color and light distribution pattern of the LED light source based on the position of surgical instruments, the tissue type of the area of ​​interest and the preset lighting standards, so as to achieve efficient fill light for the lens.

7. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the focus tracking AI laparoscope visualization method according to any one of claims 1 to 4 are implemented.

Images