Virtual training and evaluation method and device for intelligent focusing of surgical microscope

Abstract

The present invention provides a virtual training and evaluation method and device for intelligent focusing of a surgical microscope, relating to the technical field of intelligent control of surgical microscopes. The method comprises: constructing a virtual surgical scene based on real surgical video data; obtaining a current virtual microscopic texture image of a virtual surgical target captured by a virtual camera at its current position; determining a region of interest in the current virtual microscopic texture image and calculating the clarity of the region of interest; controlling the virtual camera to perform focusing motion based on the clarity of the region of interest to obtain a focus distance after the virtual camera position is updated; and calculating performance index data based on the real focus distance and the updated focus distance, thereby evaluating the focusing algorithm of the surgical microscope based on the performance index data. The present invention achieves an organic integration of virtual surgical scene depth, microscope motion control, and image quality assessment, providing a safe and reliable quantitative evaluation method for the development of intelligent focusing algorithms for surgical microscopes.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control of surgical microscopes, and in particular to a virtual training and evaluation method and device for intelligent focusing of surgical microscopes. Background Art

[0002] Surgical microscopes are key equipment in microsurgery departments such as neurosurgery. The performance of their intelligent focusing algorithms directly affects surgical safety and efficiency. However, the development of intelligent focusing algorithms for surgical microscopes faces the following technical challenges: (1) Safety risks associated with real-world surgical environment testing: surgical environment testing is required in the early stages of algorithm development, impacting patient safety; (2) Lack of quantitative evaluation criteria: Accurate focus true value data cannot be obtained, and evaluation relies on subjective judgment; (3) Difficulty in obtaining labeled data: Deep learning algorithms require a large amount of labeled data, and the cost of labeling surgical microscope videos is high.

[0003] In summary, there is currently no virtual focus training and evaluation method designed specifically for microsurgery scenarios, which severely restricts the development of intelligent focus technology for surgical microscopes. Therefore, how to develop and test intelligent focus algorithms in a safe environment, accurately quantify their performance, and reduce development costs are pressing challenges. Summary of the Invention

[0004] The present invention provides a virtual training and evaluation method and device for intelligent focusing of a surgical microscope, which are used to achieve quantitative evaluation of the performance of the intelligent focusing algorithm in a safe environment, thereby reducing development costs and risks.

[0005] The present invention provides a virtual training and evaluation method for intelligent focusing of a surgical microscope, comprising:

[0006] Constructing a virtual surgical scene based on real surgical video data; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging;

[0007] Acquire a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position;

[0008] Determining a region of interest in the current virtual microtexture image, and calculating the clarity of the region of interest;

[0009] Based on the clarity of the region of interest, controlling the virtual camera to perform focusing movement to obtain a focus distance after the virtual camera position is updated;

[0010] Based on the actual focus distance and the updated focus distance, performance indicator data is calculated to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

[0011] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the performance index data includes focus accuracy index data, time performance index data, and stability index data; the focus accuracy index data includes absolute error, relative error, and depth of field ratio; the time performance index data includes coarse focus time, fine focus time, and tracking delay time; the stability index data includes steady-state error, jitter amplitude, and convergence probability; the performance index data is calculated based on the actual focus distance and the updated focus distance, including:

[0012] Calculating the absolute value of the distance difference between the updated focus distance and the actual focus distance to obtain an absolute error;

[0013] Calculate the ratio of the absolute value of the distance difference to the actual focus distance to obtain a relative error;

[0014] Calculating a ratio of the absolute error to a depth of field range at a current magnification of the virtual camera to obtain a depth of field ratio;

[0015] When the absolute value of the distance difference is less than or equal to a coarse focus distance threshold, determining a coarse focus time;

[0016] When the absolute value of the distance difference is less than or equal to a fine focus distance threshold, determining a fine focus time;

[0017] determining a tracking delay time based on an average time lag of a focus depth change relative to a target depth change for a moving target;

[0018] Determining a steady-state error based on a trend value of the absolute value of the distance difference changing over time;

[0019] Calculate the jitter amplitude based on the step length at each moment and the average step length;

[0020] The convergence probability is determined based on the probability of achieving the target focus accuracy within a preset time.

[0021] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the real surgical video data is real surgical video data of a binocular surgical microscope; the method of constructing a virtual surgical scene based on the real surgical video data includes:

[0022] Calculate the disparity map at each moment based on the left-eye image captured by the left camera at each moment and the right-eye image captured by the right camera at each moment;

[0023] Calculating a depth coordinate of each pixel in the disparity map based on a disparity value, a focal length, and a baseline length of each pixel in the disparity map;

[0024] Calculating the three-dimensional horizontal and vertical coordinates of each pixel in the disparity map based on the depth coordinate and the two-dimensional horizontal and vertical coordinates, the focal length, and the principal point coordinates of each pixel in the disparity map;

[0025] Determine the three-dimensional coordinates of each pixel based on the three-dimensional horizontal and vertical coordinates and the depth coordinates of each pixel in the disparity map, and use the three-dimensional coordinates of each pixel as point cloud data;

[0026] The scene is reconstructed based on the point cloud data to obtain a virtual surgical scene.

[0027] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the virtual camera is a binocular virtual camera model based on the Unity engine. The internal and external parameters of the binocular virtual camera model are obtained by mapping the internal parameter matrix of a real surgical microscope and the relative position of the binocular camera to the binocular virtual camera model. The equivalent focal length of the binocular virtual camera model is calculated based on the focal length of the objective lens, the parameters of the zoom group, and the focal length of the camera adapter. The method of obtaining a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at the current position includes:

[0028] For a current magnification, generating an original virtual microscopic texture image based on an equivalent focal length of the virtual camera at the current position and internal and external parameters of the virtual camera;

[0029] The original virtual microscopic texture image is subjected to resolution adjustment and compression encoding to obtain a current virtual microscopic texture image, and the current virtual microscopic texture image is output and displayed through a data channel.

[0030] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, determining the region of interest in the current virtual microscopic texture image includes:

[0031] Calculating the texture feature response intensity based on the grayscale value of each pixel in the current virtual micro texture image;

[0032] Based on the texture feature response intensity, selecting multiple local maximum regions as candidate regions;

[0033] The texture complexity of each candidate region is calculated, and several regions with the highest texture complexity are selected as regions of interest.

[0034] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the calculation of the clarity of the region of interest includes:

[0035] Calculating a variance characteristic of the region of interest based on the grayscale value of each pixel in the region of interest and the average grayscale value of the region of interest;

[0036] Calculating the high-frequency energy distribution of the region of interest based on the Fourier transform result of the region of interest and the corresponding frequency domain representation;

[0037] A weighted calculation is performed on the variance characteristics of the region of interest and the high-frequency energy distribution of the region of interest to obtain the clarity of the region of interest.

[0038] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, controlling the virtual camera to perform focusing movement based on the clarity of the region of interest includes:

[0039] generating a focus control instruction for the region of interest based on the clarity of the region of interest;

[0040] Based on the focus control instruction, controlling the virtual camera to move according to a preset step size, determining a focus movement direction based on a difference in clarity of the virtual camera for the region of interest before and after the movement, and controlling the virtual camera to perform focus movement according to the focus movement direction;

[0041] When a maximum clarity area is detected, an adaptive step size is determined based on the reference step size, the attenuation coefficient, and the difference in clarity of the virtual camera for the area of interest before and after movement, and the virtual camera is controlled to perform focusing movement according to the adaptive step size.

[0042] The present invention also provides a virtual training and evaluation device for intelligent focusing of a surgical microscope, comprising:

[0043] A scene construction module is used to construct a virtual surgical scene based on real surgical video data; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging;

[0044] An image generation module, configured to obtain a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position;

[0045] a clarity calculation module, configured to determine a region of interest in the current virtual microtexture image and calculate the clarity of the region of interest;

[0046] A focus control module, configured to control the virtual camera to perform focus movement based on the clarity of the region of interest, and obtain a focus distance after the virtual camera position is updated;

[0047] An evaluation module is used to calculate performance indicator data based on the actual focus distance and the updated focus distance, so as to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

[0048] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the virtual training and evaluation method for intelligent focusing of a surgical microscope as described above is implemented.

[0049] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described virtual training and evaluation methods for intelligent focusing of a surgical microscope.

[0050] The present invention provides a virtual training and evaluation method and device for intelligent focusing of surgical microscopes. The method constructs a virtual surgical scene based on real surgical video data; obtains a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at its current position; determines a region of interest (ROI) in the current virtual microscopic texture image and calculates the clarity of the ROI; controls the virtual camera to perform focusing motion based on the clarity of the ROI to obtain an updated focus distance for the virtual camera position; and calculates performance index data based on the real focus distance and the updated focus distance. The method evaluates the intelligent focusing algorithm of the surgical microscope based on the performance index data. This method achieves an organic integration of virtual surgical scene depth, microscope motion control, and image quality assessment. This unified evaluation framework overcomes the limitations of traditional evaluation methods that rely on subjective judgment and provides a reliable quantitative evaluation method for the development of intelligent focusing algorithms for surgical microscopes. The present invention can obtain accurate true focus data within a safe virtual surgical environment, support objective quantitative evaluation, and reduce the development cost and risk of intelligent focusing algorithms for surgical microscopes. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 The figure is a flow chart of a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by an embodiment of the present invention.

[0053] Figure 2 This is a diagram of the composition of a virtual surgical scene provided by an embodiment of the present invention.

[0054] Figure 3 Schematic diagram of two technical routes for scene reconstruction provided by embodiments of the present invention.

[0055] Figure 4 This is a block diagram of a microscope optical simulation driven by parameter mapping provided by an embodiment of the present invention.

[0056] Figure 5 It is a schematic diagram of the interaction process provided by an embodiment of the present invention.

[0057] Figure 6 The figure is a schematic diagram of the overall process of virtual training and evaluation of intelligent focusing of a surgical microscope provided by an embodiment of the present invention.

[0058] Figure 7 It is a structural diagram of a virtual training and evaluation device for intelligent focusing of a surgical microscope provided by an embodiment of the present invention.

[0059] Figure 8 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] In the description of the embodiments of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0062] Figure 1 This is a flow chart of a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by an embodiment of the present invention. Figure 1 The embodiment of the present invention provides a virtual training and evaluation method for intelligent focusing of a surgical microscope, which may specifically include the following steps:

[0063] Step 101: construct a virtual surgical scene based on real surgical video data; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging.

[0064] It should be noted that the execution entity of the virtual training and evaluation method for intelligent focusing of a surgical microscope provided in the embodiments of the present invention can be an electronic device, a component of an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA). The non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television, ATM, or self-service machine, etc., which are not specifically limited in the embodiments of the present invention. The following description of the embodiments of the present invention uses a server as the execution entity.

[0065] In some embodiments, a binocular reconstruction solution can be used to reconstruct the scene. Specifically, a microscope equipped with a binocular real-time surgical video recording system can be used to obtain a calibrated binocular surgical microscope surgical video sequence and its camera intrinsic and extrinsic parameters to construct a virtual surgical scene. Although this solution requires relatively stringent conditions for obtaining real surgical video data, it can ensure that the reconstructed scene is consistent with the actual surgical environment to the greatest extent possible.

[0066] In some embodiments, a monocular simulation solution can be used for scene reconstruction. Specifically, ordinary 2D surgical video recording data can be combined with a standard 3D model to quickly construct a virtual surgical scene with realistic textures, which is highly flexible.

[0067] It should be noted that surgical scene reconstruction can be performed using either binocular reconstruction or monocular simulation, with flexible selection based on actual application requirements and data conditions. For algorithm verification requiring geometric precision and the availability of a binocular digital surgical microscope, binocular reconstruction is preferred for scene reconstruction. For situations requiring rapid construction of diverse test scenarios, monocular simulation can be used. A hybrid of binocular and monocular simulation solutions can ensure accurate reconstruction of key areas while enabling flexible scene expansion.

[0068] Figure 2 This is a diagram of the composition of the virtual surgical scene provided by the embodiment of the present invention. Figure 2The constructed virtual surgical scene can include a virtual surgical target with realistic texture projection and a virtual camera that simulates the imaging of a binocular microscope (i.e., a real surgical microscope). The virtual camera can be controlled for translation, rotation, and focusing along the optical axis, thereby outputting a virtual binocular imaging image of the virtual microscope during the motion control process.

[0069] The embodiments of the present invention construct a virtual surgical scene strictly based on real surgical video data. By collecting, analyzing and reconstructing the real surgical video data, it can ensure a high degree of consistency in the image texture of the virtual surgical environment and the actual surgical scene, realizing simulation based on real data. It is the key foundation for ensuring the practicality and reliability of the virtual training and evaluation method for intelligent focusing of surgical microscopes.

[0070] Moreover, the virtual environment constructed based on real surgical data can not only ensure the reliability of training and evaluation results, but also fully inherit the complexity and diversity of real surgical scenarios, thereby ensuring the effectiveness of the algorithm in practical applications. It is of great significance to ensure surgical safety, improve surgical efficiency, and promote the innovation of intelligent surgical equipment.

[0071] Step 102 : Acquire a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at the current position.

[0072] A virtual microscopic texture image may refer to a surface texture map of a virtual surgical target captured by a virtual camera. In an embodiment of the present invention, after constructing a virtual surgical scene, an image may be formed based on the characteristic that the virtual surgical scene has a texture distribution with different imaging clarity at different object distances. Specifically, a surface texture image of the observed virtual surgical target may be captured by a virtual camera. Since the depths of different areas of the observed virtual surgical target vary, some areas are within the depth of field range, while some areas have exceeded the depth of field range, resulting in the area being out of focus. Therefore, textures at different depths may have different clarity.

[0073] In an embodiment of the present invention, the virtual microscopic texture images captured by the virtual camera (different virtual microscopic texture images can reflect the characteristic of "texture distribution of virtual surgical scenes with different imaging clarity at different object distances") are further used for local area clarity identification and subsequent focus control, thereby achieving an organic unity among the depth of the virtual surgical scene, microscope motion control, and image quality assessment.

[0074] In some embodiments, the optical parameters (focal length, depth of field, magnification, etc.) and internal and external parameters of a real surgical microscope can be mapped to a virtual camera model, so that for the current magnification, based on the equivalent focal length and current internal and external parameters of the virtual camera at the current position, a current virtual microscopic texture image is generated, thereby achieving realistic imaging characteristic simulation.

[0075] The embodiments of the present invention can achieve realistic imaging effects by accurately mapping the actual parameters of the microscope (including internal and external parameters and optical parameters such as focal length, depth of field, and magnification), and can also support the simulation of complex dynamic scenes such as tissue deformation and lighting changes.

[0076] Step 103: determining a region of interest in the current virtual microscopic texture image, and calculating the clarity of the region of interest.

[0077] In an embodiment of the present invention, after initializing the virtual surgical scene and obtaining the current virtual microscopic texture image captured by the virtual camera, the current virtual microscopic texture image can be preprocessed to determine the region of interest in the current virtual microscopic texture image, thereby further calculating the clarity of the region of interest.

[0078] Step 104 : Based on the clarity of the region of interest, the virtual camera is controlled to perform focusing movement to obtain a focus distance after the virtual camera position is updated.

[0079] In the embodiment of the present invention, a focus control instruction may be generated based on the clarity of the region of interest, and the virtual camera may be controlled to perform focus movement based on the focus control instruction.

[0080] In some embodiments, a standardized network communication framework can be constructed to enable bidirectional data exchange between the virtual microscope environment and external intelligent algorithms. Specifically, a dual-channel architecture can be employed: a low-latency control channel for transmitting microscope position information and focus control commands, and a high-bandwidth data channel for transmitting real-time virtual microscopic texture image data captured by the virtual camera.

[0081] Step 105 : Calculate performance index data based on the actual focus distance and the updated focus distance, so as to evaluate the focus algorithm of the surgical microscope based on the performance index data.

[0082] In an embodiment of the present invention, a complete performance evaluation system can be constructed to reflect the performance characteristics of the focusing algorithm from different angles through multi-dimensional evaluation indicators, thereby providing a quantitative basis for the optimization of the intelligent focusing algorithm of the surgical microscope.

[0083] In some embodiments, an evaluation report containing multi-dimensional focus performance indicator data can be automatically generated based on the actual focus distance and the focus distance after position update at each moment, and a time series change curve of the performance indicator can be provided, thereby facilitating algorithm developers to conduct in-depth analysis and optimization.

[0084] The embodiments of the present invention construct a virtual surgical scene based on real surgical video data; obtain a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at the current position; determine the region of interest in the current virtual microscopic texture image and calculate the clarity of the region of interest; control the virtual camera to perform focusing motion based on the clarity of the region of interest to obtain the focus distance after the virtual camera position is updated; calculate performance index data based on the real focus distance and the updated focus distance, and evaluate the intelligent focusing algorithm of the surgical microscope based on the performance index data. This achieves an organic unity between the depth of the virtual surgical scene, microscope motion control, and image quality assessment. This unified evaluation framework overcomes the limitations of traditional evaluation methods that rely on subjective judgment and provides a reliable quantitative evaluation method for the development of intelligent focusing algorithms for surgical microscopes. The present invention can obtain accurate focus true value data in a safe virtual surgical environment, support objective quantitative evaluation, and reduce the development cost and risk of intelligent focusing algorithms for surgical microscopes.

[0085] In an optional embodiment, the real surgical video data is real surgical video data recorded by a binocular surgical microscope; and constructing a virtual surgical scene based on the real surgical video data may include:

[0086] Step S11, calculating a disparity map at each moment based on the left-eye image captured at each moment by the left-eye camera and the right-eye image captured at each moment by the right-eye camera;

[0087] Step S12, calculating the depth coordinate of each pixel in the disparity map based on the disparity value, focal length and baseline length of each pixel in the disparity map;

[0088] Step S13, calculating the three-dimensional horizontal and vertical coordinates of each pixel point in the disparity map based on the depth coordinate and the two-dimensional horizontal and vertical coordinates, the focal length and the principal point coordinates of each pixel point in the disparity map;

[0089] Step S14, determining the three-dimensional coordinates of each pixel point based on the three-dimensional horizontal and vertical coordinates and the depth coordinates of each pixel point in the disparity map, and using the three-dimensional coordinates of each pixel point as point cloud data;

[0090] Step S15: reconstructing a scene based on the point cloud data to obtain a virtual surgical scene.

[0091] Figure 3Schematic diagram of two technical routes for scene reconstruction provided by embodiments of the present invention. Figure 3 One technical approach for scene reconstruction is precise reconstruction based on binocular surgical videos. In embodiments of the present invention, this approach is applicable when a complete binocular surgical video sequence is available and the microscope's optical parameters are known. The advantage of binocular reconstruction lies in its ability to accurately restore the actual geometric structure and surface texture details of the surgical scene, making it particularly suitable for verification scenarios requiring high algorithmic accuracy.

[0092] In a specific implementation, it can be assumed that the intrinsic parameter matrices of the left and right cameras are K L and K R , the relative pose transformation matrix is T. For the left and right eye image pair I at time t L (t) and I R (t), we can use the deep learning network D to calculate the disparity map using the following formula:

[0093] ; (1)

[0094] Among them, d(t) is the disparity map at time t, I L (t) is the left eye image at time t, I R (t) is the right eye image at time t, and D is the deep learning network.

[0095] Each pixel can be , the three-dimensional coordinates (X, Y, Z) corresponding to each pixel can be calculated by the following formula:

[0096] ; (2)

[0097] ; (3)

[0098] ; (4)

[0099] Where f is the focal length, B is the baseline length, The horizontal and vertical coordinates of the main point, Pixel The disparity value of .

[0100] In some embodiments, the three-dimensional coordinates of each pixel can be used as point cloud data to obtain a point cloud sequence. After the obtained point cloud sequence is aligned, an improved Poisson reconstruction algorithm can be used to generate a watertight grid model, thereby achieving scene reconstruction based on the point cloud data. The improved Poisson reconstruction algorithm can be implemented by solving the Poisson equation:

[0101] ; (5)

[0102] in, is the target implicit function, is the point cloud normal field, The scene reconstruction process can use adaptive octree partitioning, and the depth can be dynamically adjusted according to the local point cloud density to ensure the accurate expression of model details.

[0103] Continue to refer to Figure 3 In an optional embodiment, another technical route for scene reconstruction is to achieve scene reconstruction through a rapid simulation solution based on monocular video. The monocular video reconstruction solution is suitable for situations where only ordinary 2D surgical video records are available. By mapping the texture of the real surgical video onto a standard anatomical model, a realistic virtual surgical scene can be quickly constructed. The advantages of the monocular video reconstruction solution are low data acquisition threshold, high scene reconstruction efficiency, and easy simulation of scene changes; the disadvantage is that the reconstructed geometric structure may differ from the actual surgical scene.

[0104] In a specific implementation, a suitable anatomical structure model M may be first selected from a standard model library. std , converting it to a UV-unwrapped mesh M uv , ensuring that seams and distortions are minimized. For the surgical video sequence V(t), the projection matrix of each frame of the real surgical video relative to the model can be estimated by feature matching and PnP solution:

[0105] ; (6)

[0106] Among them, K is the camera internal parameter, is the estimated camera pose.

[0107] Texture mapping can adopt a multi-frame weighted fusion strategy:

[0108] ; (7)

[0109] in, is the visibility indicator function, is the 3D point corresponding to the UV coordinate, and π is the projection function.

[0110] The weight function w(t) can comprehensively consider multiple factors such as viewing angle and clarity:

[0111] ; (8)

[0112] in, is the angle between the line of sight and the surface normal, I Ω (t) is the clarity measure of the local image area, To ensure the integrity of texture mapping, interactive supplementation and optimization can be supported to obtain the effective texture area ( ).

[0113] In an optional embodiment, the virtual camera is a binocular virtual camera model based on the Unity engine. The internal and external parameters of the binocular virtual camera model are obtained by mapping the internal parameter matrix of a real surgical microscope and the relative position of the binocular camera to the binocular virtual camera model. The equivalent focal length of the binocular virtual camera model is calculated based on the objective lens focal length, the zoom group parameters, and the camera adapter focal length. Acquiring a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at the current position may specifically include:

[0114] Step S21 , for the current magnification, based on the equivalent focal length of the virtual camera at the current position and the internal and external parameters of the virtual camera, generating an original virtual micro texture image;

[0115] Step S22 , performing resolution adjustment and compression encoding on the original virtual microscopic texture image to obtain a current virtual microscopic texture image, and outputting and displaying the current virtual microscopic texture image through a data channel.

[0116] The optical system of a surgical microscope typically consists of multiple optical components: the objective lens provides base magnification, the zoom lens enables continuous zooming, and the beam splitter system generates a dual optical path. In a digital surgical microscope, the zoom lens is connected to a high-sensitivity digital camera via an optical beam splitter and camera adapter. This system also provides direct observation for the surgeon through the eyepieces. Each camera has its own intrinsic parameters (including focal length, principal point, and sensor size), while the relative position of the two cameras (i.e., extrinsic parameters) is determined by the microscope's optical structure. This binocular camera system can be calibrated to obtain its precise intrinsic and extrinsic parameters. Although the optical path of a real surgical microscope is highly complex, the final imaging result can be seen as equivalent to two standard cameras with specific parameters.

[0117] Therefore, the embodiment of the present invention realizes imaging simulation through a binocular virtual camera model based on the Unity engine, which is conducive to realizing high-fidelity simulation of the surgical microscope optical system in a virtual environment.

[0118] Figure 4 This is a block diagram of a microscope optical simulation driven by parameter mapping provided by an embodiment of the present invention. Figure 4 In an embodiment of the present invention, realistic imaging characteristic simulation can be achieved by mapping the parameters of an actual surgical microscope to a binocular virtual camera model based on the Unity engine.

[0119] In some embodiments, a precise simulation framework for microscope optical systems can be constructed within the Unity engine. Based on the Unity engine's physical camera model, the present invention can simulate key optical properties of a microscope, including working distance, magnification, depth of field, and more. The present invention can employ a dual physical camera configuration to correspond to the left and right optical paths of the microscope. The relative position (baseline distance) and relative pose (convergence angle) of the two virtual cameras can be flexibly adjusted: through camera calibration, the intrinsic parameter matrix of the real microscope and the relative pose of the binoculars are obtained, and these parameters are mapped to the Unity engine's physical camera model.

[0120] The embodiments of the present invention utilize computer graphics and virtual reality technologies to construct a sophisticated surgical microscope imaging optical model (i.e., a binocular virtual camera model based on the Unity engine) in a commercial interactive 3D rendering engine. This also provides a semi-automated 3D reconstruction and digitization pipeline for surgical scenes, enabling rapid integration of widely acquired real surgical data from diverse sources into a highly realistic virtual environment. This allows for the integration of a full-process tool chain from scene reconstruction and algorithm development to performance evaluation, forming a closed "data-algorithm-evaluation" loop.

[0121] In some embodiments, to obtain the best stereoscopic visual experience, the present invention supports optimization of the binocular baseline distance: by conducting actual observation tests on a medical 3D display, combined with the comfortable parallax range of the human eye, the most suitable baseline distance and convergence angle configuration are determined, thereby avoiding the problem of insufficient stereoscopic perception caused by a too small baseline and the problem of visual fatigue or difficulty in binocular fusion caused by an too large baseline.

[0122] In terms of optical parameter mapping, the present invention can adjust the working distance and magnification according to the actual parameter range of the microscope, and achieve depth of field control at different magnifications by adjusting the aperture parameters of the physical camera. The equivalent focal length can be calculated based on the microscope optical transfer equation: it is determined by the focal length of the objective lens, the parameters of the zoom group, and the focal length of the camera adapter. Through parameter mapping, the embodiments of the present invention can ensure that the imaging characteristics in the virtual environment accurately reflect the optical characteristics of the actual microscope system.

[0123] In a specific implementation, taking an advanced surgical microscope such as the Zeiss Kinevo 900 as an example, the present invention can approximately simulate the continuous zoom process within its nominal working distance range. For any given magnification, the equivalent focal length at this time can be calculated based on the optical parameters of the microscope. , the equivalent focal length can be given by the objective focal length , the current magnification of the zoom group and the focal length of the camera adapter In the Unity engine, by setting parameters such as the focal length, sensor size, and field of view of the virtual camera model, a field of view similar to that of an actual microscope can be achieved. Furthermore, the aperture size can be adjusted based on the current magnification, simulating the varying depth of field at different magnifications.

[0124] In the embodiment of the present invention, each parameter of the virtual camera model can be accurately configured according to the factory parameter table of the actual microscope, thereby ensuring that the imaging effect in the virtual environment is highly consistent with the real surgical environment.

[0125] In actual use, the present invention can support the overall six-degree-of-freedom posture transformation of binocular cameras that maintain an unchanged relative position, thereby realizing stereoscopic observation under different observation angles. The focusing process can be simulated by synchronously translating the two cameras along the optical axis. The present invention can use Unity's high-definition rendering pipeline to realize the simulation of main lighting, and can support coaxial lighting configuration. By adjusting the focal plane position of the virtual camera and combining the depth of field parameters, the present invention can realize accurate simulation of the focusing effect, and can provide real-time visualization function of the focal plane position, thereby facilitating algorithm development and debugging. All parameters can be controlled and queried in real time through a unified configuration interface, and appropriate function extension interfaces are reserved to support further function extensions including special imaging modes.

[0126] In this embodiment of the present invention, by building a standardized network communication framework, bidirectional data exchange between the virtual microscope environment and external intelligent algorithms can be achieved. A dual-channel architecture can be used: a low-latency control channel is responsible for transmitting microscope position and focus control commands, and a second high-bandwidth data channel is responsible for transmitting real-time image data from the binocular camera.

[0127] In some embodiments, a reliable transmission protocol based on TCP / IP (Transmission Control Protocol / Internet Protocol) can be used in the control channel design to ensure the integrity and real-time performance of control instructions. Control instructions can be encapsulated in a unified message format and can include motion control instructions (such as translation and rotation) and focus control instructions (such as depth adjustment and speed control). Each instruction contains three basic fields: operation type, parameter axis, and motion amount. The instruction can be mapped to the corresponding microscope control parameters through a message parser. To ensure the real-time and continuity of control, an asynchronous processing mechanism can be used to decouple the instruction queue and execution process to avoid the impact of network communication delays on control performance.

[0128] In some embodiments, in the implementation of the data channel, an efficient image acquisition and encoding mechanism can be designed to meet the high-bandwidth transmission requirements of binocular image data. The present invention can first directly obtain the original virtual microtexture image data from Unity's physical camera render buffer, adjust the resolution and compress the original virtual microtexture image data to obtain the current virtual microtexture image, thereby reducing the data transmission load while ensuring image quality.

[0129] Taking into account the usage characteristics of surgical microscopes, the present invention implements a dual-frame rate adaptive mechanism: a lower image transmission frequency is used when the microscope is in a stationary state to save system resources, and it automatically switches to a high frame rate mode when performing posture adjustment or focusing operations to ensure the continuity of image feedback during the control process.

[0130] In some embodiments, the present invention implements a comprehensive resource management mechanism to ensure system stability during long-term operation. By establishing a texture buffer pool and reusing image data buffers, frequent memory allocation and release operations can be avoided. Furthermore, the present invention can include communication status monitoring and automatic recovery mechanisms, automatically reestablishing connections when network anomalies are detected and maintaining the basic state of the microscope system during reconnection.

[0131] The interface design of this embodiment fully considers the practical needs of developing intelligent focusing algorithms for surgical microscopes. Through standardized communication interfaces and efficient data transmission mechanisms, it provides a reliable platform for rapid algorithm verification and iterative optimization. The scalable design of this invention also supports the future integration of more intelligent algorithms and control strategies, providing a flexible experimental environment for the research and development of intelligent microscope control.

[0132] The embodiments of the present invention provide a standardized development and testing framework: providing a unified data interface, real-time debugging tools and evaluation indicators, thereby supporting the rapid iteration and optimization of intelligent algorithms.

[0133] In an optional embodiment, determining the region of interest in the virtual microtexture image may specifically include:

[0134] Step S31, calculating the texture feature response intensity based on the grayscale value of each pixel in the current virtual microscopic texture image;

[0135] Step S32, selecting a plurality of local maximum regions as candidate regions based on the texture feature response strength;

[0136] Step S33 : Calculate the texture complexity of each candidate region, and select several regions with the highest texture complexity as regions of interest.

[0137] In an embodiment of the present invention, by establishing a mapping relationship between the depth information of the virtual surgical scene and the actual focusing process of the microscope, an accurate evaluation of the intelligent focusing algorithm can be achieved. The present invention establishes a complete data interaction link between the virtual surgical scene, microscope image simulation, focus calculation and evaluation. In a virtual environment, the present invention can obtain the precise depth value of any point in the virtual surgical scene. , when the microscope camera system (i.e. virtual camera model) moves along the optical axis The new focal plane position With the initial position There is a linear relationship between them: .

[0138] In an embodiment of the present invention, the virtual surgical scene can be initialized, and the current virtual microscopic texture image of the virtual surgical target captured by the virtual camera can be obtained. The current virtual microscopic texture image can be preprocessed, and the local feature distribution in the preprocessed image can be calculated.

[0139] In a specific implementation, for the current virtual microscopic texture image, the texture feature response intensity can be calculated based on the grayscale value of each pixel point. The calculation formula can be as follows:

[0140] ; (9)

[0141] Where I is the image grayscale value, Ω is the image area, R(Ω) is the texture feature response intensity, is the pixel point in the image area Ω.

[0142] In a specific implementation, based on the feature response strength, N local maximum regions can be selected as candidate regions. To ensure the representativeness of the selected regions, the texture complexity of each candidate region can be further calculated, and several regions with the highest texture complexity can be selected as the regions of interest for focus evaluation. The calculation formula for texture complexity can be as follows:

[0143] ; (10)

[0144] Among them, C(Ω) is the texture complexity of the image area Ω, I is the image grayscale value, is the pixel point in the image area Ω.

[0145] In an optional embodiment, calculating the clarity of the region of interest may specifically include:

[0146] Step S41, calculating the variance characteristics of the region of interest based on the grayscale value of each pixel in the region of interest and the average grayscale value of the region of interest;

[0147] Step S42, calculating the high-frequency energy distribution of the region of interest based on the Fourier transform result of the region of interest and the corresponding frequency domain representation;

[0148] Step S43 : performing weighted calculation on the variance characteristics of the region of interest and the high-frequency energy distribution of the region of interest to obtain the clarity of the region of interest.

[0149] In a specific implementation, for the selected region of interest Ω, a multi-scale clarity measurement method can be used. First, the variance characteristics of the region Ω can be calculated in the spatial domain:

[0150] ; (11)

[0151] in, is the average gray value of the region Ω, I is the image gray value, is the pixel point in the image area Ω, S var (Ω) is the variance characteristic of region Ω.

[0152] Compute high-frequency energy distribution in the frequency domain:

[0153] ; (12)

[0154] Among them, I Ω is the grayscale value of the image Ω, is the Fourier transform of region Ω, ω is the frequency, ω0 is the high frequency cutoff threshold, S freq (Ω) is the high-frequency energy distribution in region Ω.

[0155] Combining the two metrics, we can get the final clarity evaluation result of the region of interest:

[0156] ; (13)

[0157] Among them, S var (Ω) is the variance characteristic of region Ω, S freq (Ω) is the high-frequency energy distribution in region Ω, S total (Ω) is the clarity of the region Ω; α is the weight coefficient, which can be adjusted according to the specific application scenario.

[0158] In an optional embodiment, controlling the virtual camera to perform focusing movement based on the clarity of the region of interest may specifically include:

[0159] Step S51, generating a focus control instruction for the region of interest based on the clarity of the region of interest;

[0160] Step S52: Based on the focus control instruction, controlling the virtual camera to move according to a preset step length, determining a focus movement direction based on a difference in clarity of the virtual camera for the region of interest before and after the movement, and controlling the virtual camera to perform focus movement in the focus movement direction;

[0161] Step S53, when a maximum clarity area is detected, an adaptive step size is determined based on the reference step size, the attenuation coefficient, and the difference in clarity of the virtual camera for the area of interest before and after movement, and the virtual camera is controlled to perform focusing movement according to the adaptive step size.

[0162] In an embodiment of the present invention, based on the clarity evaluation result of the region of interest, a hierarchical control strategy may be adopted to generate a focus control instruction for the region of interest to control the movement of the virtual camera.

[0163] In the coarse focus stage, the virtual camera can be controlled to move according to a preset step size. Based on the difference in clarity of the virtual camera for the area of interest before and after the movement, the focus movement direction is determined, and the virtual camera is controlled to focus along the focus movement direction.

[0164] In a specific implementation, in the coarse focus stage, a preset larger step size may be used. To scan:

[0165] ; (14)

[0166] In z n Get the current virtual micro texture image at the position, and calculate the S corresponding to the current virtual micro texture image total (Ω n ); Detective movement △z coarse distance, obtain a new virtual micro texture image, and calculate the clarity S of the new virtual micro texture image collected by the virtual camera model at the new position total (Ω n+1 ); According to sign(S total (Ω n+1 )-S total (Ω n ))Determine the final focus movement direction, that is, if the clarity is improved (the difference is positive), maintain the current movement direction for focus movement; if the clarity is reduced (the difference is negative), move in the opposite direction.

[0167] In the fine focus stage, when the maximum clarity area is detected, the adaptive step size can be determined based on the reference step size, the attenuation coefficient and the difference in clarity of the area of interest before and after the virtual camera moves, and the virtual camera can be controlled to focus according to the adaptive step size.

[0168] In a specific implementation, when a sharpness maximum area is detected, the fine focus stage can be automatically entered, using an adaptive step size:

[0169] ; (15)

[0170] in, is the benchmark step length, is the attenuation coefficient, △z fine By dynamically adjusting the step size, the present invention can improve the focusing speed while ensuring the convergence accuracy.

[0171] In an optional embodiment, the performance index data includes focus accuracy index data, time performance index data, and stability index data; the focus accuracy index data includes absolute error, relative error, and depth of field ratio; the time performance index data includes coarse focus time, fine focus time, and tracking delay time; the stability index data includes steady-state error, jitter amplitude, and convergence probability; the calculation of the performance index data based on the actual focus distance and the updated focus distance may specifically include:

[0172] Step S61, calculating the absolute value of the distance difference between the updated focus distance and the actual focus distance to obtain an absolute error;

[0173] Step S62, calculating the ratio of the absolute value of the distance difference to the actual focus distance to obtain a relative error;

[0174] Step S63, calculating the ratio of the absolute error to the depth of field range at the current magnification of the virtual camera to obtain a depth of field ratio;

[0175] Step S64, when the absolute value of the distance difference is less than or equal to the coarse focus distance threshold, determining the coarse focus time;

[0176] Step S65, when the absolute value of the distance difference is less than or equal to the fine focus distance threshold, determining the fine focus time;

[0177] Step S66, determining a tracking delay time based on an average time lag of a focus depth change relative to a target depth change for the moving target;

[0178] Step S67, determining a steady-state error based on a trend value of the absolute value of the distance difference changing over time;

[0179] Step S68, calculating the jitter amplitude based on the step length at each moment and the average step length;

[0180] Step S69 : determining the convergence probability based on the probability of achieving the target focus accuracy within a preset time.

[0181] The embodiment of the present invention constructs a complete performance evaluation system, which may include the following indicators:

[0182] a) Focus accuracy indicators:

[0183] Absolute error: ; Among them, d focus is the updated focus distance, d target is the actual focus distance;

[0184] Relative error: ; Among them, d focus is the updated focus distance, d target is the actual focus distance;

[0185] Depth of Field Ratio: , where DOF is the depth of field at the current magnification, E abs is the absolute error.

[0186] b) Time performance indicators:

[0187] Coarse focus time : From startup to first rough focus ( ) time required; of which, is the rough focus requirement range (i.e. exist This difference is within the allowable range);

[0188] Accurate focus time : From coarse focus to precise focus ( ,in, , The required range for precise focusing is exist This difference is within the allowable range) the additional time required;

[0189] Tracking Latency : For moving targets, the average time lag between the focus depth change and the target depth change.

[0190] c) Stability indicators:

[0191] Steady-state error: ;

[0192] Among them, d focus (t) is the updated focus distance at time t, d target (t) is the actual focus distance at time t;

[0193] Jitter amplitude: ;

[0194] in, is the average moving distance of the virtual camera model along the optical axis, z i is the moving distance of the virtual camera model along the optical axis in the i-th step, and N is the number of moving steps.

[0195] Convergence probability: the probability of reaching the target accuracy within a specified time.

[0196] Figure 5 This is a schematic diagram of the interaction process provided by an embodiment of the present invention. Figure 5 In some embodiments, after constructing the virtual surgical scene, a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at the current position can be obtained; a region of interest in the current virtual microscopic texture image is determined, and the clarity of the region of interest is calculated; based on the clarity of the region of interest, a focus instruction for the region of interest is generated (updating direction, step control), and the virtual camera is controlled to perform focusing movement based on the focus instruction, thereby updating the virtual camera position; after the virtual camera position is updated, an updated virtual microscopic texture image can be generated, and an updated focus distance can be obtained.

[0197] The actual focus distance of the area of interest can be obtained, and based on the actual focus distance and the updated focus distance, multi-dimensional performance index data (including focus accuracy index, time performance index and stability index, etc.) can be calculated, so as to evaluate the intelligent focus algorithm of the surgical microscope based on the multi-dimensional performance index data.

[0198] Figure 6 This is a schematic diagram of the overall process of virtual training and evaluation of intelligent focusing of surgical microscopes provided by an embodiment of the present invention. Figure 6 In some embodiments, the virtual surgical scene may be initialized first, a virtual microscopic texture image of the virtual surgical target may be acquired by a virtual camera to implement microscopic imaging simulation, and the clarity of the region of interest in the virtual microscopic texture image may be calculated.

[0199] When the clarity of the region of interest reaches its peak, determine whether the current object distance is within the depth of field range. If so, calculate multi-dimensional performance indicator data (including focus accuracy indicator, time performance indicator, stability indicator, etc.), and then evaluate the surgical microscope intelligent focus algorithm based on the multi-dimensional performance indicator data. If the current object distance is not within the depth of field range, the focus accuracy is evaluated after calculating the distance error.

[0200] When the clarity of the region of interest is not at a peak value, the virtual camera position may be updated, and the virtual micro texture image may be updated until the clarity of the region of interest in the updated virtual micro texture image reaches a peak value.

[0201] Compared with existing evaluation methods, the embodiments of the present invention achieve full parameter controllability of the focusing process and quantifiability of evaluation indicators; by obtaining accurate depth information in a virtual environment, the present invention breaks through the limitations of traditional evaluation methods that rely on subjective judgment, and provides a reliable evaluation method for the development of intelligent focusing algorithms; at the same time, the present invention supports batch automated testing, which can quickly accumulate a large amount of algorithm performance data, significantly improving the efficiency of algorithm optimization.

[0202] In summary, the embodiments of the present invention have the following advantages:

[0203] (1) Provide an accurate focus algorithm evaluation benchmark: Based on the depth information of the Unity engine, the focus true value is accurately obtained, breaking through the limitations of traditional reliance on subjective evaluation; a quantitative indicator system including focus accuracy, time performance, and system stability is established; and repeatable verification and objective comparison of algorithm performance are supported.

[0204] (2) Achieve efficient algorithm development and verification: Reduce the time for single-round algorithm verification from hours to minutes, and support 24-hour automated testing; eliminate equipment debugging and site preparation time, and improve R&D efficiency; support parameterized scenario construction, facilitating rapid testing of algorithm performance under different working conditions.

[0205] (3) Ensure scene authenticity and system reliability: Reconstruct scenes based on real surgical video data to ensure that the image characteristics of the virtual environment are highly consistent with the actual surgery; accurately map the optical parameters of the microscope (focal length, depth of field, magnification, etc.) to achieve realistic imaging effects; support the simulation of complex dynamic scenes such as tissue deformation and lighting changes.

[0206] (4) Significantly reduce development costs and risks: Algorithm development can be carried out without relying on physical microscope equipment; avoid the risk of equipment damage and operational safety hazards; and reduce data collection and annotation costs.

[0207] The virtual training and evaluation device for intelligent focusing of a surgical microscope provided by the present invention is described below. The virtual training and evaluation device for intelligent focusing of a surgical microscope described below and the virtual training and evaluation method for intelligent focusing of a surgical microscope described above can be used in correspondence with each other.

[0208] Figure 7 Schematic diagram of the structure of the virtual training and evaluation device for intelligent focusing of surgical microscope provided by the embodiment of the present invention. Figure 7 The embodiment of the present invention provides a virtual training and evaluation device for intelligent focusing of a surgical microscope, which may specifically include the following modules:

[0209] A scene construction module 710 is configured to construct a virtual surgical scene based on real surgical video data; the virtual surgical scene includes at least a virtual surgical target and a virtual camera for simulating real surgical microscope imaging;

[0210] An image generation module 720 is configured to obtain a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position;

[0211] a clarity calculation module 730 , configured to determine a region of interest in the current virtual microtexture image and calculate the clarity of the region of interest;

[0212] A focus control module 740 is configured to control the virtual camera to perform focus movement based on the clarity of the region of interest, and obtain a focus distance after the virtual camera position is updated;

[0213] The evaluation module 750 is configured to calculate performance indicator data based on the actual focus distance and the updated focus distance, so as to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

[0214] The embodiments of the present invention construct a virtual surgical scene based on real surgical video data; obtain a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at its current position to determine the region of interest in the current virtual microscopic texture image and calculate the clarity of the region of interest; based on the clarity of the region of interest, control the virtual camera to perform focusing motion to obtain the updated focus distance of the virtual camera position; calculate performance index data based on the real focus distance and the updated focus distance, and evaluate the intelligent focus algorithm of the surgical microscope based on the performance index data. This achieves an organic unity between the depth of the virtual surgical scene, microscope motion control, and image quality assessment. This unified evaluation framework overcomes the limitations of traditional evaluation methods that rely on subjective judgment and can provide a reliable quantitative evaluation method for the development of intelligent focus algorithms for surgical microscopes. The present invention can obtain accurate focus true value data in a safe virtual surgical environment, support objective quantitative evaluation, and reduce the development cost and risk of intelligent focus algorithms for surgical microscopes.

[0215] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840. The processor 810, the communications interface 820, and the memory 830 communicate with each other via the communications bus 840. The processor 810 may invoke logic instructions in the memory 830 to execute a virtual training and evaluation method for intelligent focusing of a surgical microscope. The method includes: constructing a virtual surgical scene based on real surgical video data; the virtual surgical scene includes at least a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; obtaining a current virtual microtexture image of the virtual surgical target captured by the virtual camera at a current position; determining a region of interest in the current virtual microtexture image and calculating the clarity of the region of interest; controlling the virtual camera to perform focusing movement based on the clarity of the region of interest to obtain an updated focus distance of the virtual camera position; and calculating performance indicator data based on the real focus distance and the updated focus distance, so as to evaluate the focusing algorithm of the surgical microscope based on the performance indicator data.

[0216] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0217] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the above-mentioned methods, the method including: constructing a virtual surgical scene based on real surgical video data; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; obtaining a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position; determining a region of interest in the current virtual microscopic texture image and calculating the clarity of the region of interest; based on the clarity of the region of interest, controlling the virtual camera to perform focusing movement to obtain a focus distance after the virtual camera position is updated; and calculating performance index data based on the real focus distance and the updated focus distance, so as to evaluate the focusing algorithm of the surgical microscope based on the performance index data.

[0218] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the above-mentioned methods, the method comprising: constructing a virtual surgical scene based on real surgical video data; the virtual surgical scene at least including a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; obtaining a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position; determining a region of interest in the current virtual microscopic texture image, and calculating the clarity of the region of interest; based on the clarity of the region of interest, controlling the virtual camera to perform focusing movement to obtain a focus distance after the virtual camera position is updated; and calculating performance index data based on the real focus distance and the updated focus distance, so as to evaluate the focusing algorithm of the surgical microscope based on the performance index data.

[0219] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0220] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A virtual training and evaluation method for intelligent focusing of a surgical microscope, characterized in that: include: Construct a virtual surgical scene based on real surgical video data; The virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; The virtual camera is obtained by mapping the optical parameters and internal and external parameters of a real surgical microscope to a virtual camera model, and is used to perform focusing movement in response to a focus control instruction of a focus algorithm for the surgical microscope; the focus algorithm for the surgical microscope is a focus control algorithm for controlling the focusing movement of the virtual camera; The precise depth value of any point in the virtual surgical scene can be obtained; Acquire a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position; Determining a region of interest in the current virtual microtexture image, and calculating the clarity of the region of interest; Based on the clarity of the region of interest, controlling the virtual camera to perform focusing movement to obtain a focus distance after the virtual camera position is updated; Calculating performance indicator data based on the actual focus distance and the updated focus distance, so as to evaluate a focus algorithm for the surgical microscope based on the performance indicator data; The actual focus distance is used as an evaluation benchmark to evaluate the focus algorithm for the surgical microscope.

2. The method according to claim 1, characterized in that The performance index data includes focus accuracy index data, time performance index data and stability index data; the focus accuracy index data includes absolute error, relative error and depth of field ratio, the time performance index data includes coarse focus time, fine focus time and tracking delay time, and the stability index data includes steady-state error, jitter amplitude and convergence probability; The calculating of performance indicator data based on the actual focus distance and the updated focus distance includes: Calculating the absolute value of the distance difference between the updated focus distance and the actual focus distance to obtain an absolute error; Calculate the ratio of the absolute value of the distance difference to the actual focus distance to obtain a relative error; Calculating a ratio of the absolute error to a depth of field range at a current magnification of the virtual camera to obtain a depth of field ratio; When the absolute value of the distance difference is less than or equal to a coarse focus distance threshold, determining a coarse focus time; When the absolute value of the distance difference is less than or equal to a fine focus distance threshold, determining a fine focus time; determining a tracking delay time based on an average time lag of a focus depth change relative to a target depth change for a moving target; Determining a steady-state error based on a trend value of the absolute value of the distance difference changing over time; Calculate the jitter amplitude based on the step length at each moment and the average step length; The convergence probability is determined based on the probability of achieving the target focus accuracy within a preset time.

3. The method according to claim 1, characterized in that The real surgical video data is real surgical video data of a binocular surgical microscope; The construction of a virtual surgical scene based on real surgical video data includes: Calculate the disparity map at each moment based on the left-eye image captured by the left camera at each moment and the right-eye image captured by the right camera at each moment; Calculating a depth coordinate of each pixel in the disparity map based on a disparity value, a focal length, and a baseline length of each pixel in the disparity map; Calculating the three-dimensional horizontal and vertical coordinates of each pixel in the disparity map based on the depth coordinate and the two-dimensional horizontal and vertical coordinates, the focal length, and the principal point coordinates of each pixel in the disparity map; Determine the three-dimensional coordinates of each pixel based on the three-dimensional horizontal and vertical coordinates and the depth coordinates of each pixel in the disparity map, and use the three-dimensional coordinates of each pixel as point cloud data; The scene is reconstructed based on the point cloud data to obtain a virtual surgical scene.

4. The method according to claim 1, wherein The virtual camera is a binocular virtual camera model based on the Unity engine. The internal and external parameters of the binocular virtual camera model are obtained by mapping the internal parameter matrix of the real surgical microscope and the relative position of the binocular camera to the binocular virtual camera model. The equivalent focal length of the binocular virtual camera model is calculated based on the focal length of the objective lens, the zoom group parameters, and the focal length of the camera adapter. The acquiring of a current virtual microscopic texture image of the virtual surgical target acquired by the virtual camera at the current position includes: For a current magnification, generating an original virtual microscopic texture image based on an equivalent focal length of the virtual camera at the current position and internal and external parameters of the virtual camera; The original virtual microscopic texture image is subjected to resolution adjustment and compression encoding to obtain a current virtual microscopic texture image, and the current virtual microscopic texture image is output and displayed through a data channel.

5. The method according to claim 1, wherein The determining of the region of interest in the current virtual microtexture image includes: Calculating the texture feature response intensity based on the grayscale value of each pixel in the current virtual micro texture image; Based on the texture feature response intensity, selecting multiple local maximum regions as candidate regions; The texture complexity of each candidate region is calculated, and several regions with the highest texture complexity are selected as regions of interest.

6. The method according to claim 1, characterized in that The calculating the clarity of the region of interest includes: Calculating a variance characteristic of the region of interest based on the grayscale value of each pixel in the region of interest and the average grayscale value of the region of interest; Calculating the high-frequency energy distribution of the region of interest based on the Fourier transform result of the region of interest and the corresponding frequency domain representation; A weighted calculation is performed on the variance characteristics of the region of interest and the high-frequency energy distribution of the region of interest to obtain the clarity of the region of interest.

7. The method according to claim 1, characterized in that The controlling the virtual camera to perform focusing movement based on the clarity of the region of interest includes: generating a focus control instruction for the region of interest based on the clarity of the region of interest; Based on the focus control instruction, controlling the virtual camera to move according to a preset step size, determining a focus movement direction based on a difference in clarity of the virtual camera for the region of interest before and after the movement, and controlling the virtual camera to perform focus movement according to the focus movement direction; When a maximum clarity area is detected, an adaptive step size is determined based on the reference step size, the attenuation coefficient, and the difference in clarity of the virtual camera for the area of interest before and after movement, and the virtual camera is controlled to perform focusing movement according to the adaptive step size.

8. A virtual training and evaluation device for intelligent focusing of a surgical microscope, characterized in that: include: A scene construction module is used to construct a virtual surgical scene based on real surgical video data; The virtual surgical scene includes at least a virtual surgical target and a virtual camera for simulating imaging using a real surgical microscope; the virtual camera is obtained by mapping the optical parameters and internal and external parameters of the real surgical microscope to a virtual camera model, and is configured to perform focusing movements in response to focus control instructions of a focus algorithm for the surgical microscope; the focus algorithm for the surgical microscope is a focus control algorithm for controlling the focusing movements of the virtual camera; The precise depth value of any point in the virtual surgical scene can be obtained; An image generation module, configured to obtain a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position; a clarity calculation module, configured to determine a region of interest in the current virtual microtexture image and calculate the clarity of the region of interest; A focus control module, configured to control the virtual camera to perform focus movement based on the clarity of the region of interest, and obtain a focus distance after the virtual camera position is updated; an evaluation module, configured to calculate performance indicator data based on the actual focus distance and the updated focus distance, so as to evaluate a focus algorithm for the surgical microscope based on the performance indicator data; The actual focus distance is used as an evaluation benchmark to evaluate the focus algorithm for the surgical microscope.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the virtual training and evaluation method for intelligent focusing of a surgical microscope as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the virtual training and evaluation method for intelligent focusing of a surgical microscope as claimed in any one of claims 1 to 7 is implemented.

Images