A method and system for adjusting the position of a microscope fundus surgery image

Abstract

The application provides a microscope fundus surgery image position adjustment method and system, and relates to the technical field of computers, and comprises the following steps: acquiring an original surgery video image sequence collected by a camera in real time; performing surgery scene classification according to the original surgery video image sequence to obtain a judgment signal; performing action region detection processing according to the judgment signal to obtain preliminary action region position information; performing region extraction according to the preliminary action region position information to obtain circumscribed rectangle position information of a surgery region contour; performing adjustment judgment according to the circumscribed rectangle position information to obtain an adjustment direction instruction; and performing position mapping and output according to the adjustment direction instruction, wherein the direction and distance of moving the surgery region to the center in the image plane are mapped into parameters for deflecting an optical reflection element on a two-dimensional plane to obtain adjustment parameters for driving the optical element. The application realizes automatic tracking and centering of the surgery region in the camera imaging picture.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method and system for adjusting the position of microscopic fundus surgical images. Background Technology

[0002] In microsurgical procedures, real-time video imaging and observation systems have become crucial tools for assisting surgeons in precise operations and supporting intraoperative teaching and collaboration. However, camera images are typically 16:9 rectangles, while the surgeon's view through the eyepiece is a circular field of view within the microscope tube. In fundus surgery targeting the posterior segment of the eye (such as the retina), because the surgical area is located inside the eyeball and relies on fiber optics for local illumination, the rectangular image captured by the camera usually only covers a portion of the circular field of view visible through the eyepiece. This results in many critical surgical sites located at the periphery of the field of view not being effectively imaged and recorded. To address this issue, current technologies primarily rely on non-surgical personnel manually adjusting the position of the reflector in the video optical interface during surgery to attempt to move the surgical area to the center of the image. However, this method has significant drawbacks: because fundus surgery requires rapid and continuous operations around the eyeball, and the surgical area changes rapidly, manual adjustments suffer from severe lag and inaccuracy, making it difficult to reliably ensure that all surgical actions are captured completely in real time, thus limiting the real-time, complete presentation and observation of the surgical footage.

[0003] Based on the shortcomings of the existing technology, there is an urgent need for a method and system for adjusting the position of microscopic fundus surgical images. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for adjusting the position of fundus surgical images under a microscope, thereby improving the aforementioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0005] In a first aspect, this application provides a method for adjusting the position of a fundus surgical image under a microscope, comprising:

[0006] Acquire the sequence of raw surgical video images captured in real time by the camera;

[0007] Scene features are extracted based on the original surgical video image sequence to determine whether the preset automatic adjustment triggering conditions are met, and a judgment signal is obtained.

[0008] Based on the determination signal, the action region detection process is performed. By comparing the pixel differences between consecutive image frames and locating the changed areas, preliminary action region location information is obtained.

[0009] Based on the preliminary action area location information, region extraction is performed. The current image frame is segmented into foreground and background to distinguish between the fiber-illuminated area and the unilluminated black background area. The contour of the largest continuous area is extracted to obtain the position information of the bounding rectangle of the surgical area contour.

[0010] Adjustment is determined based on the position information of the circumscribed rectangle. By calculating the distance relationship between the boundary of the circumscribed rectangle and the overall boundary of the image, it is determined whether the surgical area is at the edge of the image, and adjustment direction instructions are obtained.

[0011] The position is mapped and output according to the adjustment direction command. By mapping the direction and distance of the surgical area moving towards the center in the image plane to the parameters that control the deflection of the optical reflection element in the two-dimensional plane, the adjustment parameters for driving the optical element are obtained. The aforementioned steps are repeatedly executed based on the adjusted real-time image feedback until the surgical area is located in the preset central area.

[0012] Secondly, this application also provides a system for adjusting the position of microscopic fundus surgical images, comprising:

[0013] The acquisition module is used to acquire the sequence of raw surgical video images captured in real time by the camera;

[0014] The classification module is used to extract scene features based on the original surgical video image sequence to determine whether the preset automatic adjustment trigger conditions are met, and to obtain a judgment signal.

[0015] The detection module is used to perform action region detection processing based on the determination signal, and obtain preliminary action region location information by comparing the pixel differences between consecutive image frames and locating the changed areas.

[0016] The extraction module is used to extract regions based on the preliminary action area location information. It distinguishes between the optical fiber illuminated area and the unilluminated black background area by segmenting the foreground and background of the current image frame, and extracts the outline of the largest continuous region to obtain the position information of the bounding rectangle of the surgical area outline.

[0017] The determination module is used to make adjustment determination based on the position information of the circumscribed rectangle. By calculating the distance relationship between the boundary of the circumscribed rectangle and the overall boundary of the image, it determines whether the surgical area is at the edge of the image and obtains the adjustment direction instruction.

[0018] The output module is used to perform position mapping and output according to the adjustment direction command. By mapping the direction and distance of moving the surgical area towards the center in the image plane to the parameters for controlling the deflection of the optical reflection element in the two-dimensional plane, the adjustment parameters for driving the optical element are obtained. Based on the real-time image feedback after adjustment, the aforementioned steps are executed cyclically until the surgical area is located in the preset central area.

[0019] The beneficial effects of this invention are as follows:

[0020] This invention automatically identifies surgical scenes in the posterior segment of the eye and locates the surgical area by analyzing surgical video image sequences in real time. It then generates adjustment parameters to drive the deflection of optical reflective elements, achieving automatic tracking and centering of the surgical area within the camera's imaging frame. This invention increases the effective imaging utilization rate from 55% to nearly 100% (for the surgical region of interest) and significantly reduces the time cost associated with manual adjustments during surgery. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic flowchart illustrating a method for adjusting the position of a fundus surgical image under a microscope, as described in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the structure of a microscopic fundus surgery image position adjustment system as described in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the surgical area and the imaging area.

[0025] Figure 4 This is a schematic diagram illustrating the adjustment method of existing technology.

[0026] The diagram is labeled as follows: 901, Acquisition Module; 902, Classification Module; 903, Detection Module; 904, Extraction Module; 905, Judgment Module; 906, Output Module. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0028] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] In retinal surgery, existing microscopic surgical video systems consist of a video optical interface, a camera, and a video display and recording workstation. The video optical interface is mounted on the microscope's beam splitter, and the camera is mounted on the video optical interface, transmitting the captured images to the recording workstation for display, recording, and storage. However, it suffers from a core technical flaw: the camera captures a 16:9 rectangular image (e.g., a 1920...). 1080p HD, 3840p A 2160 4K resolution camera cannot completely cover the circular field of view observed through a microscope eyepiece; the resulting 16:9 rectangular area only occupies about 55% of the entire circular field of view. In anterior segment ophthalmic surgery, the surgical area is relatively fixed; for example, cataract surgery is concentrated within the limbus, and glaucoma surgery is concentrated in a fixed area of ​​the sclera, so the surgical area is generally within the imaging area. However, in retinal surgery, the surgical area is the entire interior of the eyeball, and many ophthalmic lesions are located in the periphery. Fiber optic illumination is used during surgery, but the illuminated area is small, and the non-surgical areas of the fundus are completely black. In this case, the camera image can only capture the overlapping portion of the illuminated area and the camera imaging area. What the doctor observes through the microscope eyepiece is a complete, circular field of view, such as... Figure 3 In this context, "Area 1: Eyepiece Field of View" refers to the area illuminated by the eyepiece. When surgical instruments move synchronously with the fiber optic illumination to treat lesions such as those affecting the peripheral retina, the illuminated "Area 3: The surgical area of ​​the fundus illuminated by the fiber optic at a given time" rapidly and continuously changes position within the circular field of view of the eyepiece. However, "Area 2: The Camera Imaging Area," captured by the fixed camera, is a relatively static rectangular image. For example... Figure 3As shown, at any given moment, the critical surgical area (green area) may fall mostly or even entirely outside this fixed blue rectangular imaging area, causing the video feed displayed in real-time to the surgical team or used for guidance to fail to fully capture the ongoing core procedure. Existing systems rely on... Figure 4 The manual adjustment mechanism shown involves non-surgical personnel visually adjusting the angle of the "rotatable mirror" in real time, thereby changing the light path and "pulling back" the illuminated green surgical area to the blue rectangular imaging area. The device used in this application adds an electrically adjustable mirror and an electric control device to the traditional optical interface. The electric control device includes two independent stepper motors, responsible for driving the mirror to move along the X and Y directions respectively. Precise stepping operations achieve fine-tuning of the mirror's orientation, thereby changing the propagation path of the microscope light and causing the eyepiece's field of view to move relative to the imaging area in all directions.

[0030] Example 1:

[0031] This embodiment provides a method for adjusting the position of microscopic fundus surgical images.

[0032] See Figure 1 The figure shows that the method includes steps S100 to S600.

[0033] Step S100: Obtain the original surgical video image sequence captured in real time by the camera;

[0034] It is understood that, in step S100, acquiring the raw surgical video image sequence captured in real time by the camera refers to continuously receiving the raw image data stream captured by the camera, which is mounted on the microscope's video optical interface, through the optical reflection element of the optical interface. This sequence consists of temporally continuous single-frame images, each of which completely contains all the optical information received by the camera sensor at that moment. This method of directly acquiring raw stream data ensures that subsequent processing is based on the most realistic and delay-free surgical field of view, providing an accurate data foundation for the entire automatic adjustment process.

[0035] Step S200: Extract scene features based on the original surgical video image sequence to determine whether the preset automatic adjustment triggering conditions are met, and obtain a judgment signal;

[0036] The core of step S200 lies in automatically distinguishing between anterior and posterior segment surgeries based on the fundamental differences in their inherent optical characteristics. Anterior segment surgeries typically utilize a microscope light source to provide uniform global illumination, resulting in a relatively consistent brightness distribution across the entire field of view. In contrast, posterior segment fundus surgeries rely on fiber optic light guides to provide localized illumination, creating a stark contrast between small areas of bright light and large areas of dark background. This step identifies these fundamental differences in illumination patterns by analyzing the overall brightness distribution, contrast, and morphology of bright areas in the image, thereby outputting a signal indicating whether the current scenario necessitates the activation of the automatic adjustment function for a posterior segment surgery.

[0037] Step S300: Perform motion region detection processing based on the judgment signal. By comparing the pixel differences between consecutive image frames and locating the changed areas, preliminary motion region location information is obtained.

[0038] The preliminary motion region detection processing in step S300 is designed based on the dynamic changes in the surgical area during a posterior ocular surgery scene. In a dark background with localized illumination, the movement of surgical instruments or tissues is the primary source of change in the image. By continuously comparing the brightness values ​​of each pixel in two consecutive frames, clusters of pixels showing changes can be quickly identified. These changing regions directly correspond to the spatial location of the surgical operation, efficiently delineating areas of interest that may contain surgical movements from the overall picture, providing focus for subsequent fine-tuning.

[0039] Step S400: Based on the preliminary action area location information, perform region extraction. By segmenting the foreground and background of the current image frame to distinguish between the area illuminated by the optical fiber and the unilluminated black background area, extract the contour of the largest continuous area to obtain the position information of the bounding rectangle of the surgical area contour.

[0040] Step S400, region extraction, involves precise identification of the physical imaging characteristics of the surgical area under fiber optic illumination. In the binarized foreground, the actual surgical area is a relatively continuous and complete region concentratedly illuminated by the fiber optic cable, while bright spots formed by intraocular fluid reflections or optical noise are usually scattered and small in area. By extracting the largest continuous contour, the illumination area representing the core of the surgical operation can be effectively distinguished and captured, while filtering out discrete interference points, thereby accurately obtaining the circumscribed rectangle reflecting the actual size and location of the surgical area.

[0041] Step S500: Adjustment determination is made based on the position information of the circumscribed rectangle. By calculating the distance relationship between the boundary of the circumscribed rectangle and the overall boundary of the image, it is determined whether the surgical area is at the edge of the image, and the adjustment direction instruction is obtained.

[0042] The adjustment determination in step S500 aims to resolve the conflict between a fixed imaging frame and a moving surgical area. When the surgical area moves to the periphery of the field of view, its circumscribed rectangle may approach or even exceed the image boundary. This step determines whether the surgical area is in an edge state, about to be removed from the image or already partially lost, by quantifying the positional relationship between the circumscribed rectangle of the surgical area and the imaging frame boundary. An adjustment command is only generated when the area is determined to be at the edge, ensuring that the adjustment is triggered only when necessary and avoiding unnecessary frequent movements.

[0043] Step S600: Perform position mapping and output according to the adjustment direction command. By mapping the direction and distance of the surgical area moving towards the center in the image plane to the parameters for controlling the deflection of the optical reflection element in the two-dimensional plane, the adjustment parameters for driving the optical element are obtained. Based on the real-time image feedback after adjustment, the aforementioned steps are executed cyclically until the surgical area is located in the preset central area.

[0044] Step S600's position mapping and output completes the final transformation from image analysis results to physical mechanical control. The direction and distance of movement calculated in the image coordinate system and used to align the center of the surgical area with the image center are pixel-level logical instructions. Controlling the deflection of the optical reflective element requires conversion into physical parameters needed for motor drive, such as pulse count and direction. This step, through a preset mapping ratio, converts the visual offset into control parameters that can directly drive the actuator (such as a stepper motor), thereby realizing an automated "perception-judgment-control" loop.

[0045] Further, step S200 includes steps S210 to S230.

[0046] Step S210: Perform feature extraction processing on single-frame images in the original surgical video image sequence. By extracting visual features of the image at multiple levels, image feature data containing color, texture and structural information is obtained.

[0047] Step S220: Perform illumination pattern feature analysis processing based on image feature data. By analyzing the distribution pattern of light spots in the bright areas of the image, the area ratio of the bright areas to the overall field of view, and the brightness and uniformity of the background areas, the scene analysis results characterizing the illumination pattern are obtained.

[0048] Step S230: Perform surgical scene determination processing based on scene analysis results. By matching and comparing the illumination mode features with the preset anterior uniform illumination mode and posterior local fiber optic illumination mode, a determination signal for the posterior surgical scene is obtained.

[0049] Specifically, in step S210, feature extraction processing is first performed on single-frame images in the original surgical video image sequence. This process systematically analyzes the visual information contained in the image at the pixel level. Color features are extracted to perceive the color differences of different tissues, texture features are extracted to describe the surface details of biological structures such as the retina and blood vessels, and structural features are extracted to grasp the overall layout relationship between light and dark areas in the field of vision, thereby obtaining a set of image feature data that can comprehensively and multidimensionally describe the content of the frame. Step S220 then performs illumination pattern feature analysis based on this feature data. Its core technical means lies in quantitative statistical analysis of the image, including: calculating whether the geometry of the bright area (i.e. the potential surgical illumination area) is concentrated in patches or scattered in points, evaluating the relative proportion of the area of ​​the bright area to the entire image field of view, and statistically analyzing the brightness level of the background area and the uniformity of its distribution. This series of analyses aims to accurately characterize the distribution characteristics of illumination from a data perspective. Its fundamental purpose is to distinguish between the two fundamentally different optical conditions: the global illumination provided by the microscope light source for anterior eye surgery, which is usually wide-coverage and uniform in brightness, and the local illumination provided by the optical fiber for posterior eye surgery, which is concentrated in range and has an extremely dark background. This results in a scene analysis that can quantitatively characterize the illumination pattern of the current image. Step S230 finally performs surgical scene determination processing based on the scene analysis results. Its core is to match and compare the quantified illumination pattern features obtained from the above analysis with the typical feature patterns that represent "uniform illumination in front of the eye" and "local fiber optic illumination in the back of the eye" respectively, which are learned by a deep convolutional neural network model (such as ResnetV2-101) trained with a large number of samples (e.g., about 10,000 surgical scene images). By calculating the matching degree or similarity between the current scene features and these two preset patterns, the model can directly output the classification result, and finally obtain a posterior ocular surgical scene determination signal that can reliably trigger the subsequent automatic adjustment process.

[0050] Further, step S300 includes steps S310 to S330.

[0051] Step S310: Based on the determination signal, obtain the current image frame to be analyzed and its immediately preceding frame image to obtain a pair of consecutive image frames to be compared.

[0052] Step S320: Perform pixel-level difference calculation processing on consecutive image frame pairs. By comparing the brightness value difference between two frames at the same coordinate position pixel by pixel, and filtering out pixels with difference values ​​exceeding a preset threshold, a pixel difference map representing the movement of surgical instruments or tissue in a dark background is obtained.

[0053] Step S330: Perform change region aggregation processing based on the pixel difference map. Connect and aggregate adjacent difference pixels in spatial location to form continuous change regions, and calculate the bounding rectangle of each change region to obtain preliminary action region location information.

[0054] Specifically, in step S310, based on the determination signal of the posterior ocular surgical scene, the current image frame to be analyzed and its immediately preceding frame in time sequence are selected to form a set of continuous image frame pairs that are continuous in time and can be used to detect changes. To balance system resource consumption and adjustment real-time performance, this application sets a fixed image acquisition and analysis frequency: one image is acquired and analyzed every 0.5 seconds (i.e., 2 frames / second). If adjustment is required after each analysis, a movement operation is performed, and the next frame continues to repeat the above process until the surgical region of interest is completely in the center of the image, at which point the adjustment operation stops. Step S320 performs pixel-level difference calculation processing based on the continuous image frame pair, comparing and subtracting the brightness values ​​of each pixel in the same spatial coordinates in the two frames. Since the background is mostly black and stable under local fiber optic illumination during posterior segment surgery, the movement of surgical instruments or tissues causes significant brightness changes in local areas due to reflected or blocked light. By setting a reasonable difference threshold and filtering out pixels exceeding this threshold, pixel changes caused by actual movements can be efficiently and initially separated, resulting in a binary pixel difference map highlighting the inter-frame movement area. Step S330 then performs change region aggregation processing based on this pixel difference map. Its core purpose is to connect and aggregate the potentially discrete point-like difference pixels obtained in the previous step according to their spatial adjacency to form a continuous change region block corresponding to a complete surgical action (such as instrument movement or tissue traction). By calculating the bounding rectangle of each aggregated connected region, preliminary action region location information expressed in rectangular coordinates can be obtained, representing the location and range of the action. The initial location determination method is quick but relatively crude, and should only be used as a reference and auxiliary method for later determination.

[0055] Further, step S400 includes steps S410 to S430.

[0056] Step S410: Based on the preliminary action area location information and the current image frame, perform image segmentation processing. Based on the brightness threshold, divide the pixels in the image into a bright foreground area and a low-brightness background area to obtain a binarized image containing the fiber optic illumination area and the black background.

[0057] Step S420: Perform region optimization processing on the binarized image. Remove discrete small bright spots formed by intraocular fluid reflection and optical noise, and perform morphological connection and filling to obtain an optimized binary image with removed interference noise.

[0058] Step S430: Perform contour extraction and position calculation processing based on the optimized binary image. By detecting all continuous region contours in the image, select the contour with the largest area as the surgical region, and calculate the circumscribed rectangle of the contour in the image coordinate system to obtain the position information of the circumscribed rectangle of the surgical region contour.

[0059] Specifically, in step S410, image segmentation is performed based on the approximate range indicated by the preliminary action area location information and the original pixel data of the current image frame. The core principle is based on the key scene feature of posterior segment surgery under local fiber optic illumination, where there is a significant brightness difference between the illuminated tissue or instrument area and the unilluminated black background. By setting a global or adaptive brightness threshold, the brightness value of each pixel in the image is compared with the threshold, thereby clearly dividing all pixels into two categories: pixels with brightness higher than the threshold are classified as bright foreground areas (i.e., potential fiber optic illumination areas), and pixels with brightness lower than the threshold are classified as low-brightness background areas. Finally, a binary image containing only pure black (background) and pure white (foreground) pixel values ​​is obtained, achieving the initial separation of the surgical illumination area. Step S420 performs region optimization processing on this binarized image to address non-surgical interference bright spots that may be present in the initial segmentation results. In the actual surgical field of the fundus, specular reflections from intraocular fluid, vitreous hyperplasia, retinal hemorrhages, and dust or noise from the optical lens may form discrete, small-area bright pixels that are not part of the actual surgical area. This step uses morphological processing to first remove these isolated small bright spots, which are much smaller than the actual surgical illumination area. Then, it connects and fills larger bright areas that may be broken due to noise or imperfect threshold segmentation but belong to the actual surgical area, making them a more complete and coherent region. This results in an optimized binary image with most of the interference noise removed. Step S430 finally performs contour extraction and position calculation based on the optimized binary image. The process involves detecting all closed continuous region contours composed of white foreground pixels in the image. In a typical scenario of posterior segment surgery, the actual surgical operation area illuminated by concentrated optical fiber is usually the largest and most prominent continuous bright area in the field of vision. Therefore, by comparing the areas enclosed by all contours, the contour with the largest area is determined to represent the core surgical area. Finally, the smallest horizontal rectangle that can exactly enclose the largest contour in the image pixel coordinate system is calculated, i.e., its circumscribed rectangle. This provides the position information of the circumscribed rectangle of the surgical area contour, which is accurately expressed in rectangular coordinates, providing a quantitative geometric basis for subsequent position determination and adjustment.

[0060] Further, step S500 includes steps S510 to S530.

[0061] Step S510: Based on the position information of the circumscribed rectangle, obtain the boundary coordinates of the circumscribed rectangle and the overall boundary coordinates of the image to obtain the coordinate data of the rectangle and the image.

[0062] Step S520: Perform edge proximity determination processing based on coordinate data. By comparing the minimum distance between each boundary of the circumscribed rectangle and the corresponding boundary of the image, and determining whether the minimum distance is less than a preset proximity threshold, a determination result representing whether the surgical area has shifted to the edge of the image is obtained.

[0063] Step S530: Perform adjustment direction calculation processing based on the judgment result. When the circumscribed rectangle is judged to be at the edge of the image, calculate the coordinate offset between the geometric center of the circumscribed rectangle and the geometric center of the image in the horizontal and vertical directions to obtain the adjustment direction command that drives the surgical area to move towards the center of the image.

[0064] Specifically, step S510 parses the coordinates of the four sides (usually represented as left, right, top, and bottom boundaries) of the circumscribed rectangle of the surgical area contour in the image pixel coordinate system based on the circumscribed rectangle's position information. Simultaneously, it acquires the boundary coordinates of the entire camera image, thus obtaining a complete set of coordinate data describing the spatial relative positional relationship between the surgical area rectangle and the image frame rectangle. Step S520 performs edge proximity determination processing based on the coordinate data. Its logic is to comprehensively evaluate the position and size of the surgical area in the image to determine whether adjustment needs to be triggered. This step first calculates the minimum pixel distance from each boundary of the circumscribed rectangle to the corresponding boundary of the image and compares it with a preset proximity threshold. A preferred judgment logic is as follows: if the distance between any boundary of the circumscribed rectangle and the corresponding boundary of the image is less than or equal to the proximity threshold (i.e., "very close" or "coinciding"), then the surgical area is deemed to have a risk of partially or completely shifting out of the camera's imaging range and needs adjustment. Simultaneously, as a supplementary optimization, if the height of the circumscribed rectangle is greater than a specific proportion of the image height (e.g., two-thirds), it indicates that the illumination range of the surgical area is already sufficiently large, essentially covering the central area of ​​the image. In this case, even if it is close to an edge, it is determined that no adjustment is needed. Step S530 finally performs adjustment direction calculation processing based on this judgment result. Specifically, when the judgment result indicates that adjustment is needed, the system calculates the difference between the geometric center coordinates of the circumscribed rectangle of the surgical area and the geometric center coordinates of the entire image in the horizontal (X-axis) and vertical (Y-axis) directions. These two coordinate offsets directly and quantitatively indicate the direction and distance of the deviation of the surgical area's center from the image center, converting this into an adjustment direction instruction that drives the surgical area to move towards the image center (e.g., move to the left by a certain number of pixels, or move down by a certain number of pixels).

[0065] Further, step S600 includes steps S610 to S630.

[0066] Step S610: The adjustment direction command is parsed and processed. By extracting the horizontal and vertical coordinate offsets in the command, the displacement data required to align the center of the surgical area with the center of the image in the image coordinate system is obtained.

[0067] Step S620: Perform coordinate-physical quantity mapping processing based on displacement data. By converting the displacement of the image pixel coordinate system into a control quantity in units of step pulses required to drive the optical reflective element to deflect according to a preset ratio, the target deflection control parameters of the reflective element are obtained.

[0068] Step S630: Generate adjustment commands based on target deflection control parameters. By outputting the lateral and longitudinal deflection control parameters as independent drive commands, adjustment parameters are obtained to control the X and Y directions of motion in the two-dimensional plane.

[0069] Specifically, step S610 parses the adjustment direction command to interpret its spatial position correction intent. The adjustment direction command essentially includes the coordinate offsets of the surgical area center relative to the image center in the horizontal (X-axis) and vertical (Y-axis) directions. The parsing process precisely extracts these offset values ​​in these two directions, thus obtaining a set of clear displacement data in the image pixel coordinate system required to align the centers of the two components. This data indicates how many pixels need to be moved in the horizontal and vertical directions, respectively. Step S620 performs coordinate-physical quantity mapping processing based on this displacement data. This is a crucial bridge for transforming visual analysis results into physical world actions. Since pixel displacement in the image is a change in virtual coordinates, while driving the actual deflection of the optical reflective element requires controlling actuators such as stepper motors, this step uses a pre-calibrated proportional relationship to precisely convert the displacement in the pixel coordinate system into a physical control quantity, measured in step pulses, required to directly drive the reflective element to deflect at a corresponding angle in a two-dimensional plane. This yields the target deflection control parameters that the reflective element needs to achieve. Step S630 finally performs adjustment instruction generation processing based on the target deflection control parameters. Its task is to encapsulate or format the lateral (X direction) and longitudinal (Y direction) control parameters obtained in the previous step and quantized into step pulse numbers into two independent drive instructions that can be recognized and executed by the hardware drive circuit or controller. These two instructions are the final adjustment parameters used to accurately control the optical reflective element to move in the X and Y directions in the two-dimensional plane, thereby completing the closed-loop control chain from image analysis to physical drive.

[0070] The method for adjusting the position of microscopic fundus surgical images provided by this invention effectively solves the problems of imaging lag, inaccuracy, and incompleteness caused by manual adjustment in traditional fundus surgery. This method achieves real-time tracking and centering of the surgical area through an automated process. Its advantages are: First, by introducing scene automatic recognition based on a deep learning model, it can intelligently distinguish between anterior and posterior segment surgeries, ensuring that the adjustment mechanism is activated only in necessary posterior segment surgeries, thus improving the system's practicality and specificity. Second, it employs a dual positioning strategy combining "preliminary inter-frame difference positioning" and "precise image segmentation positioning," integrating edge detection and coordinate mapping logic. This optimizes computational resources (e.g., analysis at specific frequencies) while achieving stable, accurate positioning and real-time tracking of the surgical area. Finally, the core processing flow of this method can be implemented by upgrading the image processing module software of existing systems without changing the main structure of the microscope. It can be deployed on the main video system architecture, demonstrating good usability and promising prospects for widespread application.

[0071] Example 2:

[0072] like Figure 2 As shown, this embodiment provides a system for adjusting the position of a fundus surgical image under a microscope. The system includes:

[0073] The acquisition module 901 is used to acquire the original surgical video image sequence captured in real time by the camera;

[0074] The classification module 902 is used to extract scene features based on the original surgical video image sequence to determine whether the preset automatic adjustment triggering conditions are met, and to obtain a judgment signal.

[0075] The detection module 903 is used to perform action region detection processing based on the judgment signal. By comparing the pixel differences between consecutive image frames and locating the changed areas, it obtains preliminary action region location information.

[0076] The extraction module 904 is used to extract regions based on the preliminary action area location information. It distinguishes between the optical fiber illuminated area and the unilluminated black background area by segmenting the foreground and background of the current image frame, and extracts the outline of the largest continuous region to obtain the position information of the bounding rectangle of the surgical area outline.

[0077] The determination module 905 is used to make adjustment determination based on the position information of the circumscribed rectangle. By calculating the distance relationship between the boundary of the circumscribed rectangle and the overall boundary of the image, it determines whether the surgical area is at the edge of the image and obtains the adjustment direction instruction.

[0078] The output module 906 is used to perform position mapping and output according to the adjustment direction command. By mapping the direction and distance of moving the surgical area towards the center in the image plane to the parameters for controlling the deflection of the optical reflection element in the two-dimensional plane, the adjustment parameters for driving the optical element are obtained. Based on the real-time image feedback after adjustment, the aforementioned steps are executed cyclically until the surgical area is located in the preset center area.

[0079] In one specific embodiment of this application, the classification module 902 includes:

[0080] The first classification unit is used to perform feature extraction processing on single-frame images in the original surgical video image sequence. By extracting visual features of the image at multiple levels, image feature data containing color, texture and structural information is obtained.

[0081] The second classification unit is used to perform lighting pattern feature analysis based on image feature data. By analyzing the distribution pattern of light spots in the bright areas of the image, the area ratio of the bright areas to the overall field of view, and the brightness and uniformity of the background areas, scene analysis results representing the lighting pattern are obtained.

[0082] The third classification unit is used to determine the surgical scene based on the scene analysis results. It obtains the determination signal of the surgical scene behind the eye by matching and comparing the illumination pattern features with the preset uniform illumination pattern in front of the eye and the local fiber optic illumination pattern in the back of the eye.

[0083] In one specific embodiment of this application, the detection module 903 includes:

[0084] The first detection unit is used to obtain the current image frame to be analyzed and its immediately preceding frame image based on the judgment signal, so as to obtain a pair of consecutive image frames to be compared.

[0085] The second detection unit is used to perform pixel-level difference calculation processing based on consecutive image frame pairs. By comparing the brightness value difference between two frames at the same coordinate position pixel by pixel, and filtering out pixels with difference values ​​exceeding a preset threshold, a pixel difference map representing the movement of surgical instruments or tissue in a dark background is obtained.

[0086] The third detection unit is used to perform change region aggregation processing based on the pixel difference map. It connects and aggregates adjacent difference pixels in spatial location to form continuous change regions, and calculates the bounding rectangle of each change region to obtain preliminary motion region location information.

[0087] In one specific embodiment of this application, the extraction module 904 includes:

[0088] The first extraction unit is used to perform image segmentation processing based on the preliminary action area location information and the current image frame. Based on the brightness threshold, the pixels in the image are divided into a bright foreground area and a low-brightness background area to obtain a binarized image containing the fiber optic illumination area and the black background.

[0089] The second extraction unit is used to perform region optimization processing on the binarized image. By removing discrete small bright spots formed by intraocular fluid reflection and optical noise, and performing morphological connection and filling, an optimized binary image with interference noise removed is obtained.

[0090] The third extraction unit is used to perform contour extraction and position calculation processing based on the optimized binary image. By detecting all continuous region contours in the image, the contour with the largest area is selected as the surgical region, and the circumscribed rectangle of the contour in the image coordinate system is calculated to obtain the position information of the circumscribed rectangle of the surgical region contour.

[0091] In one specific embodiment of this application, the determination module 905 includes:

[0092] The first determination unit is used to obtain the boundary coordinates of the circumscribed rectangle and the overall boundary coordinates of the image based on the position information of the circumscribed rectangle, so as to obtain the coordinate data of the rectangle and the image.

[0093] The second determination unit is used to perform edge proximity determination processing based on coordinate data. By comparing the minimum distance between each boundary of the circumscribed rectangle and the corresponding boundary of the image, and determining whether the minimum distance is less than a preset proximity threshold, a determination result is obtained that characterizes whether the surgical area has shifted to the edge of the image.

[0094] The third determination unit is used to perform adjustment direction calculation processing based on the determination result. When the circumscribed rectangle is determined to be at the edge of the image, the coordinate offset between the geometric center of the circumscribed rectangle and the geometric center of the image in the horizontal and vertical directions is calculated to obtain the adjustment direction command that drives the surgical area to move towards the center of the image.

[0095] In one specific embodiment of this application, the output module 906 includes:

[0096] The first output unit is used to parse and process the adjustment direction command. By extracting the horizontal and vertical coordinate offsets in the command, it obtains the displacement data required to align the center of the surgical area with the center of the image in the image coordinate system.

[0097] The second output unit is used to perform coordinate-physical quantity mapping processing based on displacement data. By converting the displacement of the image pixel coordinate system into a control quantity in units of step pulses required to drive the optical reflective element to deflect according to a preset proportional relationship, the target deflection control parameters of the reflective element are obtained.

[0098] The third output unit is used to generate adjustment commands based on the target deflection control parameters. By outputting the lateral and longitudinal deflection control parameters as independent drive commands, adjustment parameters are obtained to control the movement in the X and Y directions in the two-dimensional plane.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for adjusting the position of a fundus surgical image under a microscope, characterized in that, include: Acquire the raw surgical video image sequence captured in real time by the camera; Scene features are extracted based on the original surgical video image sequence to determine whether the preset automatic adjustment triggering conditions are met, and a judgment signal is obtained. Based on the determination signal, the action region detection process is performed. By comparing the pixel differences between consecutive image frames and locating the changed areas, preliminary action region location information is obtained. Based on the preliminary action area location information, region extraction is performed. The current image frame is segmented into foreground and background to distinguish between the fiber-illuminated area and the unilluminated black background area. The contour of the largest continuous area is extracted to obtain the position information of the bounding rectangle of the surgical area contour. Adjustment is determined based on the position information of the circumscribed rectangle. By calculating the distance relationship between the boundary of the circumscribed rectangle and the overall boundary of the image, it is determined whether the surgical area is at the edge of the image, and adjustment direction instructions are obtained. The position is mapped and output according to the adjustment direction command. By mapping the direction and distance of the surgical area moving towards the center in the image plane to the parameters that control the deflection of the optical reflection element in the two-dimensional plane, the adjustment parameters for driving the optical element are obtained. The aforementioned steps are repeatedly executed based on the adjusted real-time image feedback until the surgical area is located in the preset central area.

2. The method for adjusting the position of a fundus surgical image under a microscope according to claim 1, characterized in that, Scene features are extracted based on the original surgical video image sequence to determine whether preset automatic adjustment trigger conditions are met, resulting in a judgment signal, including: Feature extraction processing is performed on single-frame images from the original surgical video image sequence. By extracting visual features of the images at multiple levels, image feature data containing color, texture, and structural information is obtained. Based on the image feature data, lighting pattern feature analysis is performed. By analyzing the distribution pattern of light spots in the bright areas of the image, the area ratio of the bright areas to the overall field of view, and the brightness and uniformity of the background areas, scene analysis results characterizing the lighting pattern are obtained. Based on the scene analysis results, the surgical scene determination process is performed. By matching and comparing the lighting mode features with the preset anterior uniform lighting mode and posterior local fiber optic lighting mode, the determination signal of the posterior surgical scene is obtained.

3. The method for adjusting the position of a fundus surgical image under a microscope according to claim 1, characterized in that, Based on the determination signal, action region detection processing is performed. By comparing pixel differences between consecutive image frames and locating the changed regions, preliminary action region location information is obtained, including: Based on the determination signal, the current image frame to be analyzed and its immediately preceding frame image are obtained to obtain a pair of consecutive image frames to be compared. The pixel-level difference calculation is performed on the continuous image frame pairs. By comparing the brightness value difference between the two frames at the same coordinate position pixel by pixel, and filtering out the pixels with difference values ​​exceeding a preset threshold, a pixel difference map representing the movement of surgical instruments or tissue in a dark background is obtained. The pixel difference map is used to perform change region aggregation processing. By connecting and aggregating adjacent difference pixels in spatial location to form continuous change regions, the bounding rectangle of each change region is calculated to obtain preliminary action region location information.

4. The method for adjusting the position of a fundus surgical image under a microscope according to claim 1, characterized in that, Based on the preliminary action area location information, region extraction is performed. This involves segmenting the current image frame into foreground and background to distinguish between the fiber-illuminated area and the unilluminated black background area, and extracting the contour of the largest continuous region to obtain the location information of the bounding rectangle of the surgical area contour, including: Based on the preliminary action area location information and the current image frame, image segmentation processing is performed. Based on the brightness threshold, the pixels in the image are divided into a bright foreground area and a low-brightness background area to obtain a binarized image containing the fiber optic illumination area and the black background. Based on the binarized image, region optimization processing is performed. By removing discrete small bright spots formed by intraocular fluid reflection and optical noise, and performing morphological connection and filling, an optimized binary image with interference noise removed is obtained. Based on the optimized binary image, contour extraction and position calculation are performed. By detecting all continuous region contours in the image, the contour with the largest area is selected as the surgical region, and the circumscribed rectangle of the contour in the image coordinate system is calculated to obtain the position information of the circumscribed rectangle of the surgical region contour.

5. The method for adjusting the position of a fundus surgical image under a microscope according to claim 1, characterized in that, Adjustment is determined based on the position information of the circumscribed rectangle. By calculating the distance relationship between the boundary of the circumscribed rectangle and the overall boundary of the image, it is determined whether the surgical area is at the edge of the image, and adjustment direction instructions are obtained, including: Based on the position information of the circumscribed rectangle, the boundary coordinates of the circumscribed rectangle and the overall boundary coordinates of the image are obtained, thus obtaining the coordinate data of the rectangle and the image. The edge proximity determination process is performed based on the coordinate data. By comparing the minimum distance between each boundary of the circumscribed rectangle and the corresponding boundary of the image, and determining whether the minimum distance is less than a preset proximity threshold, a determination result is obtained that indicates whether the surgical area has shifted to the edge of the image. Based on the determination result, the adjustment direction calculation process is performed. When the circumscribed rectangle is determined to be at the edge of the image, the coordinate offset between the geometric center of the circumscribed rectangle and the geometric center of the image in the horizontal and vertical directions is calculated to obtain the adjustment direction command that drives the surgical area to move towards the center of the image.

6. A system for adjusting the position of a fundus surgical image under a microscope, characterized in that, include: The acquisition module is used to acquire the sequence of raw surgical video images captured in real time by the camera; The classification module extracts scene features based on the original surgical video image sequence to determine whether the preset automatic adjustment trigger conditions are met, and obtains a judgment signal. The detection module is used to perform action region detection processing based on the determination signal, and obtain preliminary action region location information by comparing the pixel differences between consecutive image frames and locating the changed areas. The extraction module is used to extract regions based on the preliminary action area location information. It distinguishes between the optical fiber illuminated area and the unilluminated black background area by segmenting the foreground and background of the current image frame, and extracts the outline of the largest continuous region to obtain the position information of the bounding rectangle of the surgical area outline. The determination module is used to make adjustment determination based on the position information of the circumscribed rectangle. By calculating the distance relationship between the boundary of the circumscribed rectangle and the overall boundary of the image, it determines whether the surgical area is at the edge of the image and obtains the adjustment direction instruction. The output module is used to perform position mapping and output according to the adjustment direction command. By mapping the direction and distance of moving the surgical area towards the center in the image plane to the parameters for controlling the deflection of the optical reflection element in the two-dimensional plane, the adjustment parameters for driving the optical element are obtained. Based on the real-time image feedback after adjustment, the aforementioned steps are executed cyclically until the surgical area is located in the preset central area.

7. The microscopic fundus surgery image position adjustment system according to claim 6, characterized in that, The classification module includes: The first classification unit is used to perform feature extraction processing on single-frame images in the original surgical video image sequence. By extracting visual features of the image at multiple levels, image feature data containing color, texture and structural information is obtained. The second classification unit is used to perform lighting pattern feature analysis processing based on the image feature data. By analyzing the light spot distribution pattern of the bright area in the image, the area ratio of the bright area to the overall field of view, and the brightness and uniformity of the background area, the scene analysis results characterizing the lighting pattern are obtained. The third classification unit is used to perform surgical scene determination processing based on the scene analysis results. By matching and comparing the lighting pattern features with the preset anterior uniform lighting pattern and posterior local fiber optic lighting pattern, a determination signal for the posterior surgical scene is obtained.

8. The microscopic fundus surgery image position adjustment system according to claim 6, characterized in that, The detection module includes: The first detection unit is used to obtain the current image frame to be analyzed and its immediately preceding frame image according to the determination signal, so as to obtain a pair of consecutive image frames to be compared. The second detection unit is used to perform pixel-level difference calculation processing based on the continuous image frame pairs. By comparing the brightness value difference between the two frames at the same coordinate position pixel by pixel, and filtering out the pixels with difference values ​​exceeding a preset threshold, a pixel difference map representing the movement of surgical instruments or tissue in a dark background is obtained. The third detection unit is used to perform change region aggregation processing based on the pixel difference map. By connecting and aggregating adjacent difference pixels in spatial location to form continuous change regions, and calculating the bounding rectangle of each change region, preliminary action region location information is obtained.

9. The microscopic fundus surgery image position adjustment system according to claim 6, characterized in that, The extraction module includes: The first extraction unit is used to perform image segmentation processing based on the preliminary action area location information and the current image frame, and divide the pixels in the image into a bright foreground area and a low-brightness background area based on the brightness threshold, so as to obtain a binarized image containing the fiber optic illumination area and the black background. The second extraction unit is used to perform region optimization processing on the binarized image. By removing discrete small bright spots formed by intraocular fluid reflection and optical noise, and performing morphological connection and filling, an optimized binary image with interference noise removed is obtained. The third extraction unit is used to perform contour extraction and position calculation processing based on the optimized binary image. By detecting all continuous region contours in the image, the contour with the largest area is selected as the surgical region, and the circumscribed rectangle of the contour in the image coordinate system is calculated to obtain the position information of the circumscribed rectangle of the surgical region contour.

10. The microscopic fundus surgery image position adjustment system according to claim 6, characterized in that, The determination module includes: The first determination unit is used to obtain the boundary coordinates of the circumscribed rectangle and the overall boundary coordinates of the image based on the position information of the circumscribed rectangle, so as to obtain the coordinate data of the rectangle and the image. The second determination unit is used to perform edge proximity determination processing based on the coordinate data. By comparing the minimum distance between each boundary of the circumscribed rectangle and the corresponding boundary of the image, and determining whether the minimum distance is less than a preset proximity threshold, a determination result is obtained that characterizes whether the surgical area has shifted to the edge of the image. The third determination unit is used to perform adjustment direction calculation processing based on the determination result. When the circumscribed rectangle is determined to be at the edge of the image, the unit calculates the coordinate offset between the geometric center of the circumscribed rectangle and the geometric center of the image in the horizontal and vertical directions to obtain the adjustment direction command that drives the surgical area to move towards the center of the image.

Images