Virtual training and evaluation method and device for intelligent focusing of operating microscope

Abstract

The invention provides a virtual training and evaluation method and device for intelligent focusing of an operating microscope, and relates to the technical field of intelligent control of operating microscopes, and the method comprises the steps: constructing a virtual operation scene based on real operation video data; obtaining a current virtual microscopic texture image which is collected by the virtual camera at the current position and aims at the virtual operation target; determining a region of interest in the current virtual microscopic texture image, and calculating the definition of the region of interest; based on the definition of the region of interest, controlling the virtual camera to perform focusing motion to obtain a focusing distance after the position of the virtual camera is updated; and calculating performance index data based on the real focusing distance and the updated focusing distance so as to evaluate the focusing algorithm of the operating microscope based on the performance index data. According to the method, organic unification among virtual operation scene depth, microscope motion control and image quality evaluation is realized, and a safe and reliable quantitative evaluation means is provided for development of an intelligent focusing algorithm of the operation microscope.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control of surgical microscopes, and in particular to a virtual training and evaluation method and device for intelligent focusing of surgical microscopes. Background Art

[0002] Surgical microscopes are key equipment in microsurgery departments such as neurosurgery, and the performance of their intelligent focus algorithms directly affects surgical safety and efficiency. However, the development of intelligent focus algorithms for surgical microscopes has the following main technical problems: (1) There are safety risks in real surgical environment testing: surgical environment testing is required in the early stages of algorithm development, which affects patient safety; (2) There is a lack of quantitative evaluation standards: it is impossible to obtain accurate focus true value data, and the evaluation relies on subjective judgment; (3) It is difficult to obtain labeled data: deep learning algorithms require a large amount of labeled data, and the cost of surgical microscope video annotation is high.

[0003] In summary, there is currently no virtual focus training and evaluation method designed for microsurgery scenarios, which seriously restricts the development of intelligent focus technology for surgical microscopes. Therefore, how to conduct development and testing in a safe environment, accurately quantify the performance of intelligent focus algorithms, and reduce development costs are issues that need to be addressed. Summary of the invention

[0004] The present invention provides a virtual training and evaluation method and device for intelligent focusing of a surgical microscope, which are used to realize quantitative evaluation of the performance of an intelligent focusing algorithm in a safe environment, thereby reducing development costs and risks.

[0005] The present invention provides a virtual training and evaluation method for intelligent focusing of a surgical microscope, comprising: Based on real surgical video data, a virtual surgical scene is constructed; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; Acquire a current virtual microscopic texture image of the virtual surgical target acquired by the virtual camera at a current position; Determine a region of interest in the current virtual microscopic texture image, and calculate the clarity of the region of interest; Based on the clarity of the region of interest, controlling the virtual camera to perform focusing movement to obtain a focus distance after the virtual camera position is updated; Based on the actual focus distance and the updated focus distance, performance indicator data is calculated to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

[0006] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the performance index data includes focus accuracy index data, time performance index data and stability index data; the focus accuracy index data includes absolute error, relative error and depth of field ratio, the time performance index data includes coarse focus time, fine focus time and tracking delay time, and the stability index data includes steady-state error, jitter amplitude and convergence probability; the performance index data is calculated based on the real focus distance and the updated focus distance, including: Calculating the absolute value of the distance difference between the updated focus distance and the actual focus distance to obtain an absolute error; Calculate the ratio of the absolute value of the distance difference to the actual focus distance to obtain a relative error; Calculate the ratio of the absolute error to the depth of field range of the virtual camera at the current magnification to obtain a depth of field ratio; When the absolute value of the distance difference is less than or equal to the coarse focus distance threshold, determining the coarse focus time; When the absolute value of the distance difference is less than or equal to the fine focus distance threshold, determining the fine focus time; Determining a tracking delay time based on an average time lag of a focus depth change relative to a target depth change for a moving target; Determine a steady-state error based on a trend value of the absolute value of the distance difference over time; Calculate the jitter amplitude based on the step length at each moment and the average step length; Based on the probability of achieving the target focus accuracy within a preset time, the convergence probability is determined.

[0007] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the real surgical video data is real surgical video data of a binocular surgical microscope; the method of constructing a virtual surgical scene based on the real surgical video data includes: Calculate a disparity map at each moment based on a left-eye image acquired at each moment by a left-eye camera and a right-eye image acquired at each moment by a right-eye camera; Calculating a depth coordinate of each pixel in the disparity map based on a disparity value, a focal length, and a baseline length of each pixel in the disparity map; Calculate the three-dimensional horizontal and vertical coordinates of each pixel in the disparity map based on the depth coordinate and the two-dimensional horizontal and vertical coordinates, the focal length and the principal point coordinates of each pixel in the disparity map; Based on the three-dimensional horizontal and vertical coordinates and the depth coordinates of each pixel in the disparity map, determine the three-dimensional coordinates of each pixel, and use the three-dimensional coordinates of each pixel as point cloud data; The scene is reconstructed based on the point cloud data to obtain a virtual surgical scene.

[0008] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the virtual camera is a binocular virtual camera model based on the Unity engine, the internal and external parameters of the binocular virtual camera model are obtained by mapping the internal parameter matrix of a real surgical microscope and the relative position and posture of the binocular camera to the binocular virtual camera model, and the equivalent focal length of the binocular virtual camera model is calculated based on the focal length of the objective lens, the zoom group parameters and the focal length of the camera adapter; the step of obtaining a current virtual microscopic texture image for the virtual surgical target collected by the virtual camera at the current position includes: For a current magnification, based on an equivalent focal length of the virtual camera at a current position and internal and external parameters of the virtual camera, an original virtual microscopic texture image is generated; The original virtual microscopic texture image is subjected to resolution adjustment and compression encoding to obtain a current virtual microscopic texture image, and the current virtual microscopic texture image is output and displayed through a data channel.

[0009] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the determining of the region of interest in the current virtual microscopic texture image comprises: Calculating the texture feature response intensity based on the grayscale value of each pixel in the current virtual microscopic texture image; Based on the texture feature response intensity, selecting multiple local maximum regions as candidate regions; The texture complexity of each candidate region is calculated, and several regions with the highest texture complexity are selected as regions of interest.

[0010] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the calculation of the clarity of the region of interest includes: Calculating a variance feature of the region of interest based on the grayscale value of each pixel in the region of interest and the average grayscale value of the region of interest; Calculating the high-frequency energy distribution of the region of interest based on the Fourier transform result of the region of interest and the corresponding frequency domain representation; The variance characteristics of the region of interest and the high-frequency energy distribution of the region of interest are weightedly calculated to obtain the clarity of the region of interest.

[0011] According to a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the present invention, the virtual camera is controlled to perform focusing movement based on the clarity of the region of interest, including: Based on the clarity of the region of interest, generating a focus control instruction for the region of interest; Based on the focus control instruction, control the virtual camera to move according to a preset step length, determine a focus movement direction based on a difference in clarity of the virtual camera for the region of interest before and after the movement, and control the virtual camera to perform a focus movement according to the focus movement direction; When a maximum clarity area is detected, an adaptive step size is determined based on a reference step size, an attenuation coefficient, and a difference in clarity of the virtual camera for the area of ​​interest before and after movement, and the virtual camera is controlled to perform focusing movement according to the adaptive step size.

[0012] The present invention also provides a virtual training and evaluation device for intelligent focusing of a surgical microscope, comprising: A scene construction module, used to construct a virtual surgery scene based on real surgery video data; the virtual surgery scene at least includes a virtual surgery target and a virtual camera for simulating real surgery microscope imaging; An image generation module, used for acquiring a current virtual microscopic texture image of the virtual surgical target acquired by the virtual camera at a current position; A clarity calculation module, used to determine a region of interest in the current virtual microscopic texture image and calculate the clarity of the region of interest; A focus control module, used to control the virtual camera to perform focus movement based on the clarity of the region of interest, and obtain a focus distance after the virtual camera position is updated; An evaluation module is used to calculate performance indicator data based on the actual focus distance and the updated focus distance, so as to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the virtual training and evaluation method for intelligent focusing of a surgical microscope as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the virtual training and evaluation method for intelligent focusing of a surgical microscope as described in any one of the above is implemented.

[0015] The virtual training and evaluation method and device for intelligent focusing of a surgical microscope provided by the present invention construct a virtual surgical scene based on real surgical video data; obtain the current virtual microscopic texture image for the virtual surgical target collected by the virtual camera at the current position; determine the region of interest in the current virtual microscopic texture image and calculate the clarity of the region of interest; based on the clarity of the region of interest, control the virtual camera to perform focusing movement to obtain the focus distance after the virtual camera position is updated; based on the real focus distance and the updated focus distance, calculate the performance index data, and evaluate the intelligent focusing algorithm of the surgical microscope based on the performance index data, thereby realizing the organic unity of the depth of the virtual surgical scene, the motion control of the microscope, and the image quality evaluation. This unified evaluation framework breaks through the limitation of the traditional evaluation method that relies on subjective judgment, and can provide a reliable quantitative evaluation method for the development of the intelligent focusing algorithm of the surgical microscope. The present invention can obtain accurate focus true value data, support objective quantitative evaluation, and reduce the development cost and risk of the intelligent focusing algorithm of the surgical microscope in a safe virtual surgical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a virtual training and evaluation method for intelligent focusing of a surgical microscope provided in an embodiment of the present invention.

[0018] Figure 2 It is a diagram of the composition of a virtual surgical scene provided by an embodiment of the present invention.

[0019] Figure 3 Schematic diagram of two technical routes for scene reconstruction provided by embodiments of the present invention.

[0020] Figure 4 It is a block diagram of a microscope optical simulation driven by parameter mapping provided by an embodiment of the present invention.

[0021] Figure 5 It is a schematic diagram of an interaction process provided by an embodiment of the present invention.

[0022] Figure 6 It is a schematic diagram of the overall process of virtual training and evaluation of intelligent focusing of a surgical microscope provided by an embodiment of the present invention.

[0023] Figure 7It is a structural schematic diagram of a virtual training and evaluation device for intelligent focusing of a surgical microscope provided by an embodiment of the present invention.

[0024] Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In the description of the embodiments of the present invention, the terms "include", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] Figure 1 1 is a flow chart of a virtual training and evaluation method for intelligent focusing of a surgical microscope provided by an embodiment of the present invention. Figure 1 The embodiment of the present invention provides a virtual training and evaluation method for intelligent focusing of a surgical microscope, and the method may specifically include the following steps: Step 101, constructing a virtual surgical scene based on real surgical video data; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging.

[0028] It should be noted that the execution subject of the virtual training and evaluation method for intelligent focusing of a surgical microscope provided in an embodiment of the present invention may be an electronic device, a component in an electronic device, an integrated circuit or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present invention. The following embodiment of the present invention is described with a server as the execution subject.

[0029] In some embodiments, a binocular reconstruction solution can be used to reconstruct the scene. Specifically, a microscope equipped with a binocular real-time surgical video recording system can be used to obtain a calibrated binocular surgical microscope surgical video sequence and its camera internal and external parameters to construct a virtual surgical scene. Although the conditions for obtaining real surgical video data are relatively harsh, this solution can ensure the consistency of the reconstructed scene with the actual surgical environment to the greatest extent.

[0030] In some embodiments, a monocular simulation solution can be used for scene reconstruction. Specifically, ordinary 2D surgical video recording data can be combined with a standard 3D model to quickly construct a virtual surgical scene with real textures, which is highly flexible.

[0031] It should be noted that the binocular reconstruction scheme or the monocular simulation scheme can be used to reconstruct the surgical scene, and the choice can be flexibly made according to the actual application requirements and data conditions. For algorithm verification that requires geometric accuracy and has binocular digital surgical microscope equipment, the binocular reconstruction scheme can be used first for scene reconstruction; for situations where a variety of test scenes need to be quickly constructed, the monocular simulation scheme can be used for scene reconstruction; the binocular reconstruction scheme and the monocular simulation scheme can be mixed and used, so as to ensure accurate reconstruction of key areas and achieve flexible expansion of scenes.

[0032] Figure 2 is a diagram of the composition of a virtual surgical scene provided by an embodiment of the present invention. Figure 2The constructed virtual surgical scene can include a virtual surgical target with real texture projection and a virtual camera for simulating the imaging of a binocular microscope camera (i.e., a real surgical microscope). The virtual camera can be controlled by translation motion, rotation motion, and focusing motion along the optical axis, so as to output a virtual binocular imaging image of the virtual microscope camera during the motion control process.

[0033] The embodiments of the present invention strictly construct a virtual surgical scene based on real surgical video data. By collecting and analyzing real surgical video data and reconstructing the scene, a high degree of consistency in the image texture of the virtual surgical environment and the actual surgical scene can be ensured, thereby achieving simulation based on real data. This is the key basis for ensuring the practicality and reliability of the virtual training and evaluation method for intelligent focusing of surgical microscopes.

[0034] Moreover, the virtual environment constructed based on real surgical data can not only ensure the reliability of training and evaluation results, but also fully inherit the complexity and diversity of real surgical scenarios, thereby ensuring the effectiveness of the algorithm in practical applications. It is of great significance to ensure surgical safety, improve surgical efficiency, and promote the innovation of intelligent surgical equipment.

[0035] Step 102: Acquire a current virtual microscopic texture image of the virtual surgical target collected by the virtual camera at the current position.

[0036] A virtual microscopic texture image may refer to a surface texture image of a virtual surgical target acquired by a virtual camera. In an embodiment of the present invention, after constructing a virtual surgical scene, an image may be formed based on the feature that the virtual surgical scene has a texture distribution with different imaging clarity at different object distances. Specifically, a surface texture image of the observed virtual surgical target may be acquired by a virtual camera. Since the depths of different areas of the observed virtual surgical target are different, some areas are within the depth of field, and some areas have exceeded the depth of field, resulting in the area being out of focus, so textures of different depths may have different clarity.

[0037] The virtual microscopic texture images captured by the virtual camera in the embodiment of the present invention (different virtual microscopic texture images can reflect the feature of "texture distribution of virtual surgical scenes with different imaging clarity at different object distances") are further used for local area clarity identification and subsequent focus control, thereby achieving an organic unity among the depth of the virtual surgical scene, microscope motion control and image quality evaluation.

[0038] In some embodiments, the optical parameters (focal length, depth of field, magnification, etc.) and internal and external parameters of a real surgical microscope can be mapped to a virtual camera model, so that for the current magnification, based on the equivalent focal length of the virtual camera at the current position and the current internal and external parameters, a current virtual microscopic texture image is generated, thereby achieving realistic imaging characteristic simulation.

[0039] The embodiments of the present invention can achieve a realistic imaging effect by accurately mapping the actual parameters of the microscope (including internal and external parameters and optical parameters such as focal length, depth of field, and magnification), and can also support the simulation of complex dynamic scenes such as tissue deformation and illumination changes.

[0040] Step 103: determine a region of interest in the current virtual microscopic texture image, and calculate the clarity of the region of interest.

[0041] In an embodiment of the present invention, after the virtual surgical scene is initialized and the current virtual microscopic texture image captured by the virtual camera is obtained, the current virtual microscopic texture image can be preprocessed to determine the region of interest in the current virtual microscopic texture image, so as to further calculate the clarity of the region of interest.

[0042] Step 104 : Based on the clarity of the region of interest, the virtual camera is controlled to perform focusing movement to obtain a focus distance after the virtual camera position is updated.

[0043] In the embodiment of the present invention, a focus control instruction may be generated based on the clarity of the region of interest, and the virtual camera may be controlled to perform focus movement based on the focus control instruction.

[0044] In some embodiments, a standardized network communication framework can be constructed to achieve bidirectional data interaction between the virtual microscope environment and the external intelligent algorithm. Specifically, a dual-channel architecture can be adopted: a low-latency control channel is used to transmit the microscope's position information and focus control instructions, and the other high-bandwidth data channel is used to transmit real-time virtual microscopic texture image data captured by the virtual camera.

[0045] Step 105 , calculating performance indicator data based on the actual focus distance and the updated focus distance, so as to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

[0046] In the embodiment of the present invention, a complete performance evaluation system can be constructed to reflect the performance characteristics of the focusing algorithm from different angles through multi-dimensional evaluation indicators, thereby providing a quantitative basis for the optimization of the intelligent focusing algorithm of the surgical microscope.

[0047] In some embodiments, an evaluation report containing multi-dimensional focus performance indicator data can be automatically generated based on the actual focus distance and the focus distance after position update at each moment, and a time series change curve of the performance indicator can be provided, thereby facilitating algorithm developers to conduct in-depth analysis and optimization.

[0048] The embodiment of the present invention constructs a virtual surgical scene based on real surgical video data; obtains the current virtual microscopic texture image of the virtual surgical target collected by the virtual camera at the current position; determines the region of interest in the current virtual microscopic texture image and calculates the clarity of the region of interest; based on the clarity of the region of interest, controls the virtual camera to perform focusing movement to obtain the focus distance after the virtual camera position is updated; based on the real focus distance and the updated focus distance, calculates performance index data, and evaluates the intelligent focus algorithm of the surgical microscope based on the performance index data, thereby achieving an organic unity among the depth of the virtual surgical scene, microscope motion control, and image quality evaluation. This unified evaluation framework breaks through the limitation of the traditional evaluation method that relies on subjective judgment, and can provide a reliable quantitative evaluation method for the development of the intelligent focus algorithm of the surgical microscope. The present invention can obtain accurate focus true value data, support objective quantitative evaluation, and reduce the development cost and risk of the intelligent focus algorithm of the surgical microscope in a safe virtual surgical environment.

[0049] In an optional embodiment, the real surgical video data is real surgical video data of a binocular surgical microscope; and constructing a virtual surgical scene based on the real surgical video data includes, specifically, including: Step S11, calculating a disparity map at each moment based on a left-eye image acquired at each moment by a left-eye camera and a right-eye image acquired at each moment by a right-eye camera; Step S12, calculating the depth coordinate of each pixel in the disparity map based on the disparity value, focal length and baseline length of each pixel in the disparity map; Step S13, calculating the three-dimensional horizontal and vertical coordinates of each pixel point in the disparity map based on the depth coordinate and the two-dimensional horizontal and vertical coordinates, the focal length and the principal point coordinates of each pixel point in the disparity map; Step S14, determining the three-dimensional coordinates of each pixel point based on the three-dimensional horizontal and vertical coordinates and the depth coordinates of each pixel point in the disparity map, and using the three-dimensional coordinates of each pixel point as point cloud data; Step S15, reconstructing the scene based on the point cloud data to obtain a virtual surgical scene.

[0050] Figure 3 Schematic diagram of two technical routes for scene reconstruction provided by embodiments of the present invention. Figure 3One technical route for scene reconstruction is an accurate reconstruction scheme based on binocular surgical videos. In the embodiment of the present invention, the accurate reconstruction scheme based on binocular surgical videos can be applied to the situation where a complete binocular surgical video sequence has been acquired and the optical parameters of the microscope are known. The advantage of the binocular reconstruction scheme is that it can accurately restore the real geometric structure and surface texture details of the surgical scene, which is particularly suitable for verification scenarios with high requirements for algorithm accuracy.

[0051] In a specific implementation, it can be assumed that the intrinsic parameter matrices of the left and right cameras are K L and K R , the relative pose transformation matrix is ​​T. For the left and right eye image pair at time t I L (t) and I R (t), we can use the deep learning network D to calculate the disparity map using the following formula: ; (1) Among them, d(t) is the disparity map at time t, I L (t) is the left eye image at time t, I R (t) is the right eye image at time t, and D is the deep learning network.

[0052] Based on the disparity map and camera parameters, each pixel , the three-dimensional coordinates (X, Y, Z) corresponding to each pixel can be calculated by the following formula: ; (2) ; (3) ; (4) Where f is the focal length, B is the baseline length, is the horizontal and vertical coordinates of the main point, Pixel The parallax value of .

[0053] In some embodiments, the three-dimensional coordinates of each pixel point can be used as point cloud data to obtain a point cloud sequence. After the obtained point cloud sequence is registered, an improved Poisson reconstruction algorithm can be used to generate a watertight grid model, thereby realizing scene reconstruction based on the point cloud data. The improved Poisson reconstruction algorithm can be implemented by solving the Poisson equation: ; (5) in, is the target implicit function, is the point cloud normal field, is the divergence operator. The scene reconstruction process can use adaptive octree partitioning, and the depth can be dynamically adjusted according to the local point cloud density to ensure the accurate expression of model details.

[0054] Continue to refer to Figure 3 In an optional embodiment, another technical route for scene reconstruction is to achieve scene reconstruction based on a fast simulation solution based on monocular video. The monocular video reconstruction solution is suitable for situations where only ordinary 2D surgical video records are available. By mapping the real surgical video texture onto a standard anatomical model, a realistic virtual surgical scene can be quickly constructed. The advantages of the monocular video reconstruction solution are low data acquisition threshold, high scene reconstruction efficiency, and ease of simulation of scene changes; the disadvantage is that the reconstructed geometric structure may be somewhat different from the actual surgical scene.

[0055] In a specific implementation, a suitable anatomical structure model M may be first selected from a standard model library. std , converting it to a UV-unwrapped mesh M uv , ensuring minimization of seams and distortions. For the surgical video sequence V(t), the projection matrix of each frame of the real surgical video relative to the model can be estimated by feature matching and PnP solution: ; (6) Among them, K is the camera internal parameter, is the estimated camera pose.

[0056] Texture mapping can adopt a multi-frame weighted fusion strategy: ; (7) in, is the visibility indicator function, is the 3D point corresponding to the UV coordinate, and π is the projection function.

[0057] The weight function w(t) can comprehensively consider multiple factors such as viewing angle and clarity: ; (8) in, is the angle between the line of sight and the surface normal, I Ω (t) is the clarity measure of the local image area, is the required range of angle. To ensure the integrity of texture mapping, interactive supplementation and optimization can be supported to obtain the effective texture area ( ).

[0058] In an optional embodiment, the virtual camera is a binocular virtual camera model based on the Unity engine, the internal and external parameters of the binocular virtual camera model are obtained by mapping the internal parameter matrix of the real surgical microscope and the relative position of the binocular camera to the binocular virtual camera model, and the equivalent focal length of the binocular virtual camera model is calculated based on the objective lens focal length, the zoom group parameter and the camera adapter focal length; the obtaining of the current virtual microscopic texture image of the virtual surgical target collected by the virtual camera at the current position may specifically include: Step S21, for the current magnification, based on the equivalent focal length of the virtual camera at the current position and the internal and external parameters of the virtual camera, generating an original virtual microscopic texture image; Step S22, adjusting the resolution and compressing the original virtual microscopic texture image to obtain a current virtual microscopic texture image, and outputting and displaying the current virtual microscopic texture image through a data channel.

[0059] The optical system of a surgical microscope is usually composed of multiple optical elements: the objective lens group provides the basic magnification, the zoom group realizes continuous zooming, and the beam splitting system produces a dual optical path. In a digital surgical microscope, the rear end of the zoom group is connected to a highly sensitive digital camera through an optical splitter and a camera adapter, and also provides direct observation for the surgeon through the eyepiece group. Each camera has its inherent internal parameters (including focal length, principal point, photosensitive element size, etc.), while the relative position between the two cameras (i.e., external parameters) is determined by the optical structure of the microscope. This binocular camera system can obtain its precise internal and external parameters through calibration. Although the optical path of an actual surgical microscope is very complex, from the perspective of the final imaging effect, the entire system can be equivalent to two standard cameras with specific parameters.

[0060] Therefore, the embodiment of the present invention realizes imaging simulation through a binocular virtual camera model based on the Unity engine, which is conducive to realizing high-fidelity surgical microscope optical system simulation in a virtual environment.

[0061] Figure 4 is a block diagram of a microscope optical simulation driven by parameter mapping provided by an embodiment of the present invention. Figure 4 In an embodiment of the present invention, realistic imaging characteristic simulation can be achieved by mapping the parameters of an actual surgical microscope to a binocular virtual camera model based on the Unity engine.

[0062] In some embodiments, an accurate simulation framework of the microscope optical system can be constructed in the Unity engine. Based on the physical camera model of the Unity engine, the present invention can simulate the key optical properties of the microscope, including working distance, magnification, depth of field, etc. The present invention can use a dual physical camera structure to correspond to the left and right optical paths of the microscope. The relative position (baseline distance) and relative posture (convergence angle) of the two virtual cameras can be flexibly adjusted: the intrinsic parameter matrix and binocular relative posture of the real microscope are obtained through camera calibration, and these parameters are mapped to the physical camera model of the Unity engine.

[0063] The embodiment of the present invention utilizes computer graphics and virtual reality technology to build a sophisticated surgical microscope imaging optical model (i.e., a binocular virtual camera model based on the Unity engine) in a commercial interactive 3D rendering engine, while providing a semi-automated 3D reconstruction and digitization pipeline for the surgical scene, which can quickly integrate widely acquired real surgical data from different sources into a highly realistic virtual environment, thereby integrating a full-process tool chain from scene reconstruction, algorithm development to performance evaluation, forming a closed loop of "data-algorithm-evaluation".

[0064] In some embodiments, in order to obtain the best stereoscopic visual experience, the present invention supports the optimization of the binocular baseline distance: by conducting actual observation tests on a medical 3D display, combined with the comfortable parallax range of the human eye, the most suitable baseline distance and convergence angle configuration are determined, thereby avoiding the problem of insufficient stereoscopic perception due to a too small baseline and the problem of visual fatigue of the observer or difficulty in binocular fusion due to an too large baseline.

[0065] In terms of optical parameter mapping, the present invention can adjust the working distance and magnification according to the actual parameter range of the microscope, and realize depth of field control under different magnifications by adjusting the aperture parameters of the physical camera. The equivalent focal length can be calculated according to the optical transfer equation of the microscope: it is jointly determined by the focal length of the objective lens, the parameters of the zoom group, and the focal length of the camera adapter. Through parameter mapping, the embodiment of the present invention can ensure that the imaging characteristics in the virtual environment can accurately reflect the optical characteristics of the actual microscope system.

[0066] In a specific implementation, taking the advanced surgical microscope such as Zeiss Kinevo 900 as an example, the present invention can approximately simulate the continuous zoom process within its nominal working distance range. For any given magnification, the equivalent focal length at this time can be calculated based on the optical parameters of the microscope. , the equivalent focal length can be calculated from the objective lens focal length , zoom group current magnification and camera adapter focal length In the Unity engine, by setting the focal length, sensor size, and field of view of the virtual camera model, a field of view that is approximately consistent with that of an actual microscope can be achieved. At the same time, the aperture size can be adjusted according to the current magnification to simulate the depth of field variation characteristics under different magnifications.

[0067] In the embodiment of the present invention, each parameter of the virtual camera model can be accurately configured according to the factory parameter table of the actual microscope, thereby ensuring that the imaging effect in the virtual environment is highly consistent with the real surgical environment.

[0068] In actual use, the present invention can support the overall six-degree-of-freedom posture transformation of binocular cameras that maintain an unchanged relative position, thereby realizing stereoscopic observation under different observation angles. The focusing process can be simulated by synchronously translating the two cameras along the optical axis. The present invention can use Unity's high-definition rendering pipeline to realize the simulation of main lighting, and can support coaxial lighting configuration. By adjusting the focal plane position of the virtual camera and combining the depth of field parameters, the present invention can achieve accurate simulation of the focusing effect, and can provide real-time visualization function of the focal plane position, thereby facilitating algorithm development and debugging. All parameters can be controlled and queried in real time through a unified configuration interface, and appropriate function expansion interfaces are reserved to support further function expansion including special imaging modes.

[0069] In the embodiment of the present invention, by constructing a standardized network communication framework, two-way data interaction between the virtual microscope environment and the external intelligent algorithm can be achieved. A dual-channel architecture can be adopted: a low-latency control channel is responsible for transmitting the microscope's position and focus control instructions, and the other high-bandwidth data channel is responsible for transmitting the real-time image data of the binocular camera.

[0070] In some embodiments, in the control channel design, a reliable transmission protocol based on TCP / IP (Transmission Control Protocol / Internet Protocol) can be used to ensure the integrity and real-time performance of the control instructions. The control instructions can be encapsulated in a unified message format, and can include motion control instructions (such as translation, rotation) and focus control instructions (such as depth adjustment, speed control), etc. Each instruction contains three basic fields: operation type, parameter axis, and motion amount. The instructions can be mapped to the corresponding microscope control parameters through a message parser. To ensure the real-time and continuity of the control, an asynchronous processing mechanism can be used to decouple the instruction queue and the execution process to avoid the impact of network communication delays on control performance.

[0071] In some embodiments, in the implementation of the data channel, an efficient image acquisition and encoding mechanism can be designed for the high bandwidth transmission requirements of binocular image data. The present invention can first directly obtain the original virtual micro texture image data from the physical camera rendering buffer of Unity, adjust the resolution and compress the original virtual micro texture image data to obtain the current virtual micro texture image, thereby reducing the data transmission load while ensuring the image quality.

[0072] Taking into account the usage characteristics of surgical microscopes, the present invention implements a dual frame rate adaptive mechanism: a lower image transmission frequency is used when the microscope is in a stationary state to save system resources, and it automatically switches to a high frame rate mode when performing posture adjustment or focusing operations to ensure the continuity of image feedback during the control process.

[0073] In some embodiments, the present invention implements a complete resource management mechanism for the system stability requirements during long-term operation. By setting a texture buffer pool and reusing the image data buffer, frequent memory allocation and release operations can be avoided. At the same time, the present invention can include a communication status monitoring and automatic recovery mechanism, which can automatically reconnect when a network anomaly is detected and maintain the basic state of the microscope system during the reconnection.

[0074] The embodiment of the present invention takes into full account the actual needs of the development process of the intelligent focusing algorithm for surgical microscopes through interface design; through standardized communication interfaces and efficient data transmission mechanisms, a reliable support platform can be provided for rapid verification and iterative optimization of the algorithm. The scalable design of the present invention can also support the integration of more types of intelligent algorithms and control strategies in the future, providing a flexible experimental environment for the research and development of intelligent control of microscopes.

[0075] The embodiment of the present invention provides a standardized development and testing framework: providing a unified data interface, real-time debugging tools and evaluation indicators, thereby supporting rapid iteration and optimization of intelligent algorithms.

[0076] In an optional embodiment, determining the region of interest in the virtual microscopic texture image may specifically include: Step S31, calculating the texture feature response intensity based on the gray value of each pixel in the current virtual microscopic texture image; Step S32, selecting a plurality of local maximum regions as candidate regions based on the texture feature response strength; Step S33, calculating the texture complexity of each candidate region, and selecting several regions with the highest texture complexity as regions of interest.

[0077] In the embodiment of the present invention, by establishing a mapping relationship between the depth information of the virtual surgical scene and the actual focusing process of the microscope, an accurate evaluation of the intelligent focusing algorithm can be achieved. The present invention establishes a complete data interaction link between the virtual surgical scene, microscope image simulation, focus calculation and evaluation. In a virtual environment, the present invention can obtain the accurate depth value of any point in the virtual surgical scene. , when the microscope camera system (i.e., virtual camera model) moves along the optical axis The new focal plane position With initial position There is a linear relationship between them: .

[0078] In an embodiment of the present invention, the virtual surgical scene can be initialized, a current virtual microscopic texture image of the virtual surgical target captured by a virtual camera can be obtained, the current virtual microscopic texture image can be preprocessed, and the local feature distribution in the preprocessed image can be calculated.

[0079] In a specific implementation, for the current virtual microscopic texture image, the texture feature response intensity can be calculated based on the grayscale value of each pixel point, and the calculation formula can be as follows: ; (9) Where I is the gray value of the image, Ω is the image area, R(Ω) is the texture feature response intensity, is the pixel point in the image area Ω.

[0080] In a specific implementation, based on the characteristic response intensity, N local maximum regions can be selected as candidate regions. To ensure the representativeness of the selected regions, the texture complexity of each candidate region can be further calculated, and several regions with the highest texture complexity can be selected as regions of interest for focus evaluation. The calculation formula for texture complexity can be as follows: ; (10) Among them, C(Ω) is the texture complexity of the image area Ω, I is the image gray value, is the pixel point in the image area Ω.

[0081] In an optional embodiment, the calculating the clarity of the region of interest may specifically include: Step S41, calculating the variance feature of the region of interest based on the grayscale value of each pixel in the region of interest and the average grayscale value of the region of interest; Step S42, calculating the high-frequency energy distribution of the region of interest based on the Fourier transform result of the region of interest and the corresponding frequency domain representation; Step S43: performing weighted calculation on the variance feature of the region of interest and the high-frequency energy distribution of the region of interest to obtain the clarity of the region of interest.

[0082] In a specific implementation, for the selected region of interest Ω, a multi-scale clarity measurement method can be used. First, the variance feature of the region Ω can be calculated in the spatial domain: ; (11) in, is the average gray value of region Ω, I is the image gray value, is the pixel point in the image area Ω, S var (Ω) is the variance characteristic of region Ω.

[0083] Compute high-frequency energy distribution in the frequency domain: ; (12) Among them, I Ω is the gray value of the image Ω, is the Fourier transform of the region Ω, ω is the frequency, ω 0 is the high frequency cut-off threshold, S freq (Ω) is the high-frequency energy distribution in region Ω.

[0084] Combining the two metrics, we can get the final clarity evaluation result of the region of interest: ; (13) Among them, S var (Ω) is the variance characteristic of region Ω, S freq (Ω) is the high-frequency energy distribution in region Ω, S total (Ω) is the clarity of area Ω; α is the weight coefficient, which can be adjusted according to the specific application scenario.

[0085] In an optional embodiment, controlling the virtual camera to perform focusing movement based on the clarity of the region of interest may specifically include: Step S51, generating a focus control instruction for the region of interest based on the clarity of the region of interest; Step S52, based on the focus control instruction, controlling the virtual camera to move according to a preset step length, determining a focus movement direction based on a difference in clarity of the virtual camera for the region of interest before and after the movement, and controlling the virtual camera to perform a focus movement according to the focus movement direction; Step S53, when a maximum clarity area is detected, an adaptive step size is determined based on a reference step size, an attenuation coefficient, and a difference in clarity of the virtual camera for the area of ​​interest before and after movement, and the virtual camera is controlled to perform focusing movement according to the adaptive step size.

[0086] In the embodiment of the present invention, based on the clarity evaluation result of the region of interest, a hierarchical control strategy may be adopted to generate a focus control instruction for the region of interest to control the movement of the virtual camera.

[0087] In the coarse focus stage, the virtual camera can be controlled to move according to a preset step size, and the focus movement direction is determined based on the difference in clarity of the area of ​​interest before and after the movement of the virtual camera, and the virtual camera is controlled to focus in the focus movement direction.

[0088] In a specific implementation, in the coarse focusing stage, a preset larger step size may be used. To scan: ; (14) In z n Get the current virtual microscopic texture image at the position, and calculate the S corresponding to the current virtual microscopic texture image total (Ω n ); Detective movement △z coarse distance, obtain a new virtual microscopic texture image, and calculate the clarity S of the new virtual microscopic texture image collected by the virtual camera model at the new position total (Ω n+1 ); according to sign(S total (Ω n+1 )-S total (Ω n ))Determine the final focus movement direction, that is, if the clarity is improved (the difference is positive), keep the current movement direction for focus movement; if the clarity is reduced (the difference is negative), move in the opposite direction.

[0089] In the fine focus stage, when the maximum clarity area is detected, the adaptive step size can be determined based on the reference step size, the attenuation coefficient and the clarity difference of the area of ​​interest before and after the virtual camera moves, and the virtual camera can be controlled to perform focusing movement according to the adaptive step size.

[0090] In a specific implementation, when a maximum value area of ​​sharpness is detected, the fine focus stage can be automatically entered, using an adaptive step size: ; (15) in, is the base step length, is the attenuation coefficient, △z fine By dynamically adjusting the step size, the present invention can improve the focusing speed while ensuring the convergence accuracy.

[0091] In an optional embodiment, the performance index data includes focus accuracy index data, time performance index data and stability index data; the focus accuracy index data includes absolute error, relative error and depth of field ratio, the time performance index data includes coarse focus time, fine focus time and tracking delay time, and the stability index data includes steady-state error, jitter amplitude and convergence probability; the performance index data is calculated based on the real focus distance and the updated focus distance, which may specifically include: Step S61, calculating the absolute value of the distance difference between the updated focus distance and the actual focus distance to obtain an absolute error; Step S62, calculating the ratio of the absolute value of the distance difference to the actual focus distance to obtain a relative error; Step S63, calculating the ratio of the absolute error to the depth of field range of the virtual camera at the current magnification to obtain a depth of field ratio; Step S64, when the absolute value of the distance difference is less than or equal to the coarse focus distance threshold, determining the coarse focus time; Step S65, when the absolute value of the distance difference is less than or equal to the fine focus distance threshold, determining the fine focus time; Step S66, determining a tracking delay time based on an average time lag of a focus depth change relative to a target depth change for a moving target; Step S67, determining a steady-state error based on a trend value of the absolute value of the distance difference changing with time; Step S68, calculating the jitter amplitude based on the step length at each moment and the step length average; Step S69, determining the convergence probability based on the probability of reaching the target focus accuracy within a preset time.

[0092] The embodiment of the present invention constructs a complete performance evaluation system, which may include the following indicators: a) Focus accuracy indicators: Absolute error: ; where d focus is the updated focus distance, d target is the actual focusing distance; Relative error: ; where d focus is the updated focus distance, d target is the actual focusing distance; Depth of Field Ratio: , where DOF is the depth of field at the current magnification, E abs is the absolute error.

[0093] b) Time performance indicators: Coarse focus time : From startup to first rough focus ( ) time required; among which, is the rough focus requirement range (i.e. exist This difference is within the allowable range); Accurate focus time : From coarse focus to precise focus ( ,in, , The required range for precise focusing is exist This difference is within the permissible range) the additional time required; Tracking Latency : For moving targets, the average time lag of focus depth change relative to target depth change.

[0094] c) Stability indicators: Steady-state error: ; Among them, d focus (t) is the updated focus distance at time t, d target (t) is the actual focus distance at time t; Jitter Amplitude: ; in, is the average moving distance of the virtual camera model along the optical axis, z i is the moving distance of the virtual camera model along the optical axis in the i-th step, and N is the number of moving steps.

[0095] Convergence probability: the probability of reaching the target accuracy within a specified time.

[0096] Figure 5 is a schematic diagram of the interaction process provided by an embodiment of the present invention. Figure 5 In some embodiments, after constructing a virtual surgical scene, a current virtual microscopic texture image of a virtual surgical target captured by a virtual camera at a current position can be obtained; a region of interest in the current virtual microscopic texture image is determined, and the clarity of the region of interest is calculated; based on the clarity of the region of interest, a focus instruction for the region of interest is generated (updating direction, step control), and the virtual camera is controlled to perform a focus movement based on the focus instruction, thereby updating the position of the virtual camera; after the virtual camera position is updated, an updated virtual microscopic texture image can be generated, and an updated focus distance can be obtained.

[0097] The actual focus distance of the area of ​​interest can be obtained, and based on the actual focus distance and the updated focus distance, multi-dimensional performance indicator data (including focus accuracy index, time performance index and stability index, etc.) can be calculated, so as to evaluate the intelligent focus algorithm of the surgical microscope based on the multi-dimensional performance indicator data.

[0098] Figure 6 1 is a schematic diagram of the overall process of virtual training and evaluation of intelligent focusing of a surgical microscope provided by an embodiment of the present invention. Figure 6 In some embodiments, the virtual surgical scene may be initialized first, a virtual microscopic texture image of a virtual surgical target may be acquired by a virtual camera to implement microscopic imaging simulation, and the clarity of the region of interest in the virtual microscopic texture image may be calculated.

[0099] When the clarity of the area of ​​interest is at its peak, determine whether the current object distance is within the depth of field. If so, calculate multi-dimensional performance indicator data (including focus accuracy indicator, time performance indicator, stability indicator, etc.), so as to evaluate the intelligent focus algorithm of the surgical microscope based on the multi-dimensional performance indicator data; if the current object distance is not within the depth of field, perform a focus accuracy evaluation after calculating the distance error.

[0100] When the clarity of the region of interest is not at a peak value, the virtual camera position may be updated, and the virtual microscopic texture image may be updated until the clarity of the region of interest in the updated virtual microscopic texture image reaches a peak value.

[0101] Compared with the existing evaluation methods, the embodiments of the present invention achieve full parameter controllability of the focusing process and quantifiability of the evaluation indicators; by obtaining accurate depth information in a virtual environment, the present invention breaks through the limitation of traditional evaluation methods that rely on subjective judgment, and provides a reliable evaluation method for the development of intelligent focusing algorithms; at the same time, the present invention supports batch automated testing, which can quickly accumulate a large amount of algorithm performance data, significantly improving the efficiency of algorithm optimization.

[0102] In summary, the embodiments of the present invention have the following advantages: (1) Providing an accurate focus algorithm evaluation benchmark: Accurately obtain the true focus value based on the depth information of the Unity engine, breaking through the limitations of traditional reliance on subjective evaluation; establishing a quantitative indicator system that includes focus accuracy, time performance, and system stability; and supporting repeatable verification and objective comparison of algorithm performance.

[0103] (2) Achieve efficient algorithm development and verification: The time for a single round of algorithm verification is reduced from hours to minutes, supporting 24-hour automated testing; eliminating equipment debugging and site preparation time, improving R&D efficiency; supporting parameterized scenario construction, facilitating rapid testing of algorithm performance under different working conditions.

[0104] (3) Ensure scene authenticity and system reliability: Reconstruct scenes based on real surgical video data to ensure that the image features of the virtual environment are highly consistent with the actual surgery; accurately map the optical parameters of the microscope (focal length, depth of field, magnification, etc.) to achieve realistic imaging effects; support the simulation of complex dynamic scenes such as tissue deformation and lighting changes.

[0105] (4) Significantly reduce development costs and risks: algorithm development can be carried out without relying on physical microscope equipment; avoid the risk of equipment damage and operational safety hazards; and reduce data collection and labeling costs.

[0106] The virtual training and evaluation device for intelligent focusing of a surgical microscope provided by the present invention is described below. The virtual training and evaluation device for intelligent focusing of a surgical microscope described below and the virtual training and evaluation method for intelligent focusing of a surgical microscope described above can be referenced to each other.

[0107] Figure 7 Schematic diagram of the structure of a virtual training and evaluation device for intelligent focusing of a surgical microscope provided by an embodiment of the present invention. Figure 7 The embodiment of the present invention provides a virtual training and evaluation device for intelligent focusing of a surgical microscope, and the device may specifically include the following modules: A scene construction module 710 is used to construct a virtual surgery scene based on real surgery video data; the virtual surgery scene at least includes a virtual surgery target and a virtual camera for simulating real surgery microscope imaging; An image generation module 720 is used to obtain a current virtual microscopic texture image of the virtual surgical target acquired by the virtual camera at a current position; A clarity calculation module 730 is used to determine a region of interest in the current virtual microscopic texture image and calculate the clarity of the region of interest; A focus control module 740 is used to control the virtual camera to perform focus movement based on the clarity of the region of interest, and obtain a focus distance after the virtual camera position is updated; The evaluation module 750 is used to calculate performance indicator data based on the actual focus distance and the updated focus distance, so as to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

[0108] The embodiment of the present invention constructs a virtual surgical scene based on real surgical video data; obtains the current virtual microscopic texture image of the virtual surgical target collected by the virtual camera at the current position to determine the region of interest in the current virtual microscopic texture image, and calculates the clarity of the region of interest; based on the clarity of the region of interest, controls the virtual camera to perform focusing movement to obtain the focus distance after the virtual camera position is updated; based on the real focus distance and the updated focus distance, calculates performance index data, and evaluates the intelligent focus algorithm of the surgical microscope based on the performance index data, thereby achieving an organic unity among the depth of the virtual surgical scene, microscope motion control, and image quality evaluation. This unified evaluation framework breaks through the limitation of the traditional evaluation method that relies on subjective judgment, and can provide a reliable quantitative evaluation method for the development of the intelligent focus algorithm of the surgical microscope. The present invention can obtain accurate focus true value data, support objective quantitative evaluation, and reduce the development cost and risk of the intelligent focus algorithm of the surgical microscope in a safe virtual surgical environment.

[0109] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the virtual training and evaluation method of intelligent focusing of the surgical microscope, the method comprising: constructing a virtual surgical scene based on real surgical video data; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; obtaining a current virtual microscopic texture image for the virtual surgical target collected by the virtual camera at the current position; determining an area of ​​interest in the current virtual microscopic texture image, and calculating the clarity of the area of ​​interest; based on the clarity of the area of ​​interest, controlling the virtual camera to perform focusing movement, and obtaining the focus distance after the virtual camera position is updated; based on the real focus distance and the updated focus distance, calculating performance index data, so as to evaluate the focus algorithm of the surgical microscope based on the performance index data.

[0110] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0111] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the above methods, the method including: constructing a virtual surgical scene based on real surgical video data; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; obtaining a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position; determining an area of ​​interest in the current virtual microscopic texture image, and calculating the clarity of the area of ​​interest; based on the clarity of the area of ​​interest, controlling the virtual camera to perform focusing movement to obtain a focus distance after the virtual camera position is updated; and calculating performance indicator data based on the real focus distance and the updated focus distance, so as to evaluate the focusing algorithm of the surgical microscope based on the performance indicator data.

[0112] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the virtual training and evaluation method for intelligent focusing of a surgical microscope provided by the above-mentioned methods, the method comprising: constructing a virtual surgical scene based on real surgical video data; the virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; obtaining a current virtual microscopic texture image of the virtual surgical target captured by the virtual camera at a current position; determining a region of interest in the current virtual microscopic texture image, and calculating the clarity of the region of interest; based on the clarity of the region of interest, controlling the virtual camera to perform focusing movement to obtain a focus distance after the virtual camera position is updated; and calculating performance indicator data based on the real focus distance and the updated focus distance, so as to evaluate the focusing algorithm of the surgical microscope based on the performance indicator data.

[0113] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual training and evaluation method for intelligent focusing of a surgical microscope, characterized in that: include: Construct a virtual surgical scene based on real surgical video data; The virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; Acquire a current virtual microscopic texture image of the virtual surgical target acquired by the virtual camera at a current position; Determine a region of interest in the current virtual microscopic texture image, and calculate the clarity of the region of interest; Based on the clarity of the region of interest, controlling the virtual camera to perform focusing movement to obtain a focus distance after the virtual camera position is updated; Based on the actual focus distance and the updated focus distance, performance indicator data is calculated to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

2. The method according to claim 1, characterized in that The performance index data includes focus accuracy index data, time performance index data and stability index data; the focus accuracy index data includes absolute error, relative error and depth of field ratio, the time performance index data includes coarse focus time, fine focus time and tracking delay time, and the stability index data includes steady-state error, jitter amplitude and convergence probability; The calculating performance indicator data based on the actual focus distance and the updated focus distance includes: Calculating the absolute value of the distance difference between the updated focus distance and the actual focus distance to obtain an absolute error; Calculate the ratio of the absolute value of the distance difference to the actual focus distance to obtain a relative error; Calculate the ratio of the absolute error to the depth of field range of the virtual camera at the current magnification to obtain a depth of field ratio; When the absolute value of the distance difference is less than or equal to the coarse focus distance threshold, determining the coarse focus time; When the absolute value of the distance difference is less than or equal to the fine focus distance threshold, determining the fine focus time; Determining a tracking delay time based on an average time lag of a focus depth change relative to a target depth change for a moving target; Determine a steady-state error based on a trend value of the absolute value of the distance difference over time; Calculate the jitter amplitude based on the step length at each moment and the average step length; Based on the probability of achieving the target focus accuracy within a preset time, the convergence probability is determined.

3. The method according to claim 1, characterized in that The real surgical video data is real surgical video data of a binocular surgical microscope; The construction of a virtual surgical scene based on real surgical video data includes: Calculate a disparity map at each moment based on a left-eye image acquired at each moment by a left-eye camera and a right-eye image acquired at each moment by a right-eye camera; Calculating a depth coordinate of each pixel in the disparity map based on a disparity value, a focal length, and a baseline length of each pixel in the disparity map; Calculate the three-dimensional horizontal and vertical coordinates of each pixel in the disparity map based on the depth coordinate and the two-dimensional horizontal and vertical coordinates, the focal length and the principal point coordinates of each pixel in the disparity map; Based on the three-dimensional horizontal and vertical coordinates and the depth coordinates of each pixel in the disparity map, determine the three-dimensional coordinates of each pixel, and use the three-dimensional coordinates of each pixel as point cloud data; The scene is reconstructed based on the point cloud data to obtain a virtual surgical scene.

4. The method according to claim 1, characterized in that: The virtual camera is a binocular virtual camera model based on the Unity engine, the internal and external parameters of the binocular virtual camera model are obtained by mapping the internal parameter matrix of the real surgical microscope and the relative position of the binocular camera to the binocular virtual camera model, and the equivalent focal length of the binocular virtual camera model is calculated based on the focal length of the objective lens, the zoom group parameters and the focal length of the camera adapter; The step of acquiring a current virtual microscopic texture image for the virtual surgical target acquired by the virtual camera at the current position includes: For a current magnification, based on an equivalent focal length of the virtual camera at a current position and internal and external parameters of the virtual camera, an original virtual microscopic texture image is generated; The original virtual microscopic texture image is subjected to resolution adjustment and compression encoding to obtain a current virtual microscopic texture image, and the current virtual microscopic texture image is output and displayed through a data channel.

5. The method according to claim 1, characterized in that The determining of the region of interest in the current virtual microscopic texture image comprises: Calculating the texture feature response intensity based on the grayscale value of each pixel in the current virtual microscopic texture image; Based on the texture feature response intensity, selecting multiple local maximum regions as candidate regions; The texture complexity of each candidate region is calculated, and several regions with the highest texture complexity are selected as regions of interest.

6. The method according to claim 1, characterized in that The calculating the clarity of the region of interest comprises: Calculating a variance feature of the region of interest based on the grayscale value of each pixel in the region of interest and the average grayscale value of the region of interest; Calculating the high-frequency energy distribution of the region of interest based on the Fourier transform result of the region of interest and the corresponding frequency domain representation; The variance characteristics of the region of interest and the high-frequency energy distribution of the region of interest are weightedly calculated to obtain the clarity of the region of interest.

7. The method according to claim 1, characterized in that The controlling the virtual camera to perform focusing movement based on the clarity of the region of interest includes: Based on the clarity of the region of interest, generating a focus control instruction for the region of interest; Based on the focus control instruction, control the virtual camera to move according to a preset step length, determine a focus movement direction based on a difference in clarity of the virtual camera for the region of interest before and after the movement, and control the virtual camera to perform a focus movement according to the focus movement direction; When a maximum clarity area is detected, an adaptive step size is determined based on a reference step size, an attenuation coefficient, and a difference in clarity of the virtual camera for the area of ​​interest before and after movement, and the virtual camera is controlled to perform focusing movement according to the adaptive step size.

8. A virtual training and evaluation device for intelligent focusing of a surgical microscope, characterized in that: include: The scene construction module is used to construct a virtual surgical scene based on real surgical video data; The virtual surgical scene at least includes a virtual surgical target and a virtual camera for simulating real surgical microscope imaging; An image generation module, used for acquiring a current virtual microscopic texture image of the virtual surgical target acquired by the virtual camera at a current position; A clarity calculation module, used to determine a region of interest in the current virtual microscopic texture image and calculate the clarity of the region of interest; A focus control module, used to control the virtual camera to perform focus movement based on the clarity of the region of interest, and obtain a focus distance after the virtual camera position is updated; An evaluation module is used to calculate performance indicator data based on the actual focus distance and the updated focus distance, so as to evaluate the focus algorithm of the surgical microscope based on the performance indicator data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the virtual training and evaluation method for intelligent focusing of a surgical microscope as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the virtual training and evaluation method for intelligent focusing of a surgical microscope as claimed in any one of claims 1 to 7 is implemented.

Images