Lesion region visual positioning method for intelligent multi-arm microsurgical robot
By acquiring microsurgical image sequences under multiple focal planes and performing sharpness assessment and image fusion, the problem of insufficient reliability and accuracy in lesion localization was solved, and precise identification of lesion areas was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SECOND AFFILIATED HOSPITAL ZHEJIANG UNIV COLLEGE OF MEDICINE
- Filing Date
- 2026-03-10
- Publication Date
- 2026-06-09
Smart Images
Figure CN121810689B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of surgical computer-aided technology, and more specifically to a method for visual localization of lesion areas in intelligent multi-arm microsurgical robots. Background Technology
[0002] As medical technology advances towards precision and intelligence, intelligent multi-arm microsurgical robots, with their collaborative operation of multiple robotic arms, high-precision motion control, and intraoperative image processing capabilities, have been widely applied in delicate microsurgical procedures such as neurosurgery and ophthalmology. These robots assist surgeons in improving the stability of surgical procedures and enhancing visual information. Visual localization of the lesion area, as a core pre-operative step, directly affects surgical path planning and operational accuracy, and is a key technical point for ensuring surgical safety and efficacy.
[0003] In a microsurgical environment, due to the limited depth of field of high-magnification microscopes, the displayed images can only clearly show a local area of the target region. Other areas of the image are out of focus due to the limited depth of field, making it difficult to obtain a complete and clear image of the lesion and surrounding tissues. This reduces the ability to distinguish the boundary between the lesion area and normal tissue in the image, ultimately affecting the reliability and accuracy of lesion localization. Summary of the Invention
[0004] To address the technical problem of low reliability and accuracy in lesion area localization in existing technologies, the present invention aims to provide a visual localization method for lesion areas using an intelligent multi-arm microsurgical robot. The specific technical solution adopted is as follows:
[0005] Acquire a sequence of microsurgical images of the target region under multiple focal planes; the sequence of microsurgical images includes focal plane images corresponding to each of the multiple focal planes; the pixels in the focal plane images corresponding to each of the multiple focal planes are one-to-one correspondences.
[0006] For each focal plane image, the sharpness of each pixel in the focal plane image is evaluated, and the sharpness weight of each pixel is determined.
[0007] Image fusion is performed based on the sharpness weights of pixels in each focal plane image, and the lesion area in the target region is identified and located based on the generated fused image.
[0008] In one possible implementation, the method includes:
[0009] For each focal plane image, determine the mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel; the mean motion coefficient is used to characterize the overall tendency of motion blur in the focal plane image; the defocus evaluation coefficient is used to characterize the degree of blurring caused by defocusing of the pixel.
[0010] For each pixel in the focal plane image, the sharpness weight of the pixel is determined based on the mean motion coefficient of the focal plane image, the defocus evaluation coefficient of each pixel, and the pixel position of each pixel.
[0011] In one possible implementation, the method includes:
[0012] For each focal plane image, multiple transition points are selected from the focal plane image; transition points are pixels in the focal plane image located at the edge regions of different tissues or structures.
[0013] Cluster analysis based on the diffusion direction of multiple transition points yields multiple clusters; each cluster includes one or more transition points.
[0014] The mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel are determined based on the diffusion direction of the transition points in multiple clusters.
[0015] In one possible implementation, the method includes:
[0016] For each pixel in each focal plane image, the uniformity of the pixel is determined based on the gray values of the pixel and its neighboring points.
[0017] Pixels in the focal plane image whose uniformity is less than the uniformity threshold are selected as transition points.
[0018] In one possible implementation, the method includes:
[0019] For each cluster, the motion coefficient of the cluster is calculated based on the spatial clustering degree of the transition points within the cluster and the consistency of the diffusion direction.
[0020] The mean motion coefficient of the focal plane image is obtained based on the motion coefficient of each cluster in the focal plane image;
[0021] The defocus evaluation coefficient of each pixel is determined based on the motion coefficient of each cluster in the focal plane image and the diffusion direction of the transition point.
[0022] In one possible implementation, the method further includes:
[0023] For each transition point in the focal plane image, a gradient direction sequence is determined based on the gradient gray-level change and gradient direction change of the transition point and its neighboring points. The gradient direction sequence includes the transition point corresponding to each level of gradient.
[0024] The gradient direction between the transition point and the transition point corresponding to the last level gradient in the gradient direction sequence is taken as the diffusion direction of the transition point.
[0025] In one possible implementation, the method includes:
[0026] For each transition point in the focal plane image, the last gradient level in the corresponding gradient direction sequence is taken as the diffusion level of the transition point, and the diffusion evaluation coefficient of the transition point is determined based on the gradient gray value change of each level in the gradient direction sequence.
[0027] Based on the motion coefficient of the cluster where the transition point is located, the diffusion direction of each transition point in the cluster where the transition point is located, and the diffusion level and diffusion evaluation coefficient of the transition point, the defocus evaluation coefficient of the transition point is determined.
[0028] Based on the connectivity between each transition point in the focal plane image, the focal plane image is divided into multiple connected regions; each connected region includes multiple pixels.
[0029] For each connected component, the average value of the defocus evaluation coefficient of each transition point in the connected component is used as the defocus evaluation coefficient of each pixel in the connected component.
[0030] In one possible implementation, the method includes:
[0031] Based on the depth range of the target area, determine the start and end points of the acquisition path and the focal plane step size;
[0032] The electric focusing device that controls the intelligent multi-arm microsurgical robot moves from the starting point to the ending point step by step according to the focal plane step, and triggers the camera exposure at each focal plane position to capture the corresponding focal plane image.
[0033] In one possible implementation, capturing the corresponding focal plane image is triggered by real-time monitoring of the target object's physiological signals, which indicate whether the target object is in a quiescent state.
[0034] In one possible implementation, the method includes:
[0035] For pixels at the same pixel location in each focal plane image, the pixel with the highest sharpness weight at that pixel location is selected for image fusion to generate a fused image;
[0036] Based on an image segmentation model, lesion areas in the target region are identified and located from fused images.
[0037] The present invention has the following beneficial effects:
[0038] Based on the above technical solution, this application avoids the local blurring problem caused by the extremely small depth of field of a single focal plane by acquiring a sequence of microsurgical images under multiple focal planes. Then, by evaluating the sharpness of each pixel to determine the sharpness weight, it ensures that the sharpest pixel is used at each position in the fused image, effectively improving the overall sharpness of the full-focus image. Finally, the lesion area is located based on the clear fused image. Compared with the existing technical solutions, this application solves the problem of inaccurate positioning caused by local blurring in existing single focal plane images or simple stitched images, effectively improving the reliability and accuracy of lesion area positioning, and providing a reliable image data foundation for the intelligent and precise operation of intelligent multi-arm microsurgical robots. Attached Figure Description
[0039] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 This is a flowchart illustrating a method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot, provided in one embodiment of the present invention.
[0041] Figure 2 This is a flowchart illustrating another method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot, provided in one embodiment of the present invention.
[0042] Figure 3 This is a flowchart illustrating another method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot, provided in one embodiment of the present invention.
[0043] Figure 4 This is a flowchart illustrating another method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot, provided in one embodiment of the present invention.
[0044] Figure 5 This is a flowchart illustrating another method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot, provided in one embodiment of the present invention.
[0045] Figure 6 A schematic diagram of a first scene of a gradient grayscale iteration process provided in an embodiment of the present invention;
[0046] Figure 7 This is a schematic diagram of a second scenario of a gradient grayscale iteration process provided in an embodiment of the present invention. Detailed Implementation
[0047] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the lesion area visual localization method for an intelligent multi-arm microsurgical robot proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.
[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0049] In view of the technical problems of low reliability and accuracy in lesion area localization in existing technologies, this application avoids the local blurring problem caused by the extremely small depth of field of a single focal plane by acquiring a sequence of microsurgical images under multiple focal planes. Then, by evaluating the sharpness of each pixel to determine the sharpness weight, it ensures that the sharpest pixel is used at each position in the fused image, effectively improving the overall sharpness of the full-focus image. Finally, the lesion area is located based on the clear fused image. Compared with existing technical solutions, this application solves the problem of inaccurate localization caused by local blurring in existing single focal plane images or simple stitched images, effectively improving the reliability and accuracy of lesion area localization, and providing a reliable image data foundation for the intelligent and precise operation of intelligent multi-arm microsurgical robots.
[0050] The specific solution of the lesion area visual localization method for intelligent multi-arm microsurgical robots provided by the present invention will be described in detail below with reference to the accompanying drawings.
[0051] Please see Figure 1 The diagram illustrates a method flowchart for visual localization of lesion areas in an intelligent multi-arm microsurgical robot according to an embodiment of the present invention. The method includes the following steps:
[0052] Step 101: Acquire a sequence of microsurgical images of the target area under multiple focal planes.
[0053] The microsurgical image sequence includes multiple focal plane images corresponding to each focal plane. The pixels in the multiple focal plane images correspond one-to-one, meaning that the same pixel location corresponds to the same spatial location of the target area (such as a blood vessel section or tissue boundary) in different focal plane images, ensuring the effectiveness of subsequent pixel clarity comparison.
[0054] In one possible implementation, this application can determine the start point, end point, and focal plane step size of the acquisition path based on the depth range of the target area. Then, the electric focusing device of the intelligent multi-arm microsurgical robot is controlled to move gradually from the start point to the end point according to the focal plane step size, and the camera is triggered to expose at each focal plane position to capture the corresponding focal plane image.
[0055] For example, the depth range of the target area can be determined using preoperative scanning data from an intelligent multi-arm microsurgical robot. The focal plane step size can be set based on actual conditions, considering both acquisition efficiency and sharpness: a step size that is too small will increase the number of images and prolong processing time; a step size that is too large may miss sharp pixels. For example, the focal plane step size can be set to 1 / 2 to 1 / 3 of the microscope depth of field, such as 0.5 μm to 2 μm.
[0056] In one example, this application can control the electric focusing device to move step by step, triggering camera exposure at each focal plane to capture an image, obtaining a series of images covering the entire target depth. Then, the captured image sequence is transmitted in real time to computer memory for temporary storage via a high-speed interface, preparing for subsequent processing.
[0057] In some embodiments, capturing the corresponding focal plane image is triggered based on real-time monitoring of the physiological signals of the target object, which are used to indicate whether the target object is in a quiescent state.
[0058] The resting period refers to the period during which physiological activities (such as heartbeat and respiration) result in the least range of tissue movement.
[0059] For example, physiological signals can be obtained by an electrocardiogram (ECG) monitor (monitoring heartbeat) or a respiratory sensor (monitoring respiration). For instance, during the interval between the R waves in an ECG signal (RR interval), the amplitude of cardiac motion is minimal and can be considered as the cardiac resting period; during the interval between the end of expiration and the beginning of inspiration in a respiratory signal, the amplitude of chest and abdominal motion is minimal and can be considered as the respiratory resting period.
[0060] When the target object is detected to enter the stationary phase, a trigger signal is generated to control the camera to complete the exposure within the stationary phase (such as triggering the exposure within the middle 100ms of the RR interval), so as to avoid image blurring caused by tissue movement and further improve the clarity of the acquired image.
[0061] Step 102: For each focal plane image, perform a sharpness evaluation on each pixel in the focal plane image and determine the sharpness weight of each pixel.
[0062] The sharpness weight is used to characterize the sharpness of a pixel under the current focal plane. The larger the sharpness weight, the sharper the pixel is under the current focal plane, and the more suitable it is as a pixel source for that pixel position in the fused image.
[0063] For images acquired at different focal planes, the areas of sharpness vary due to the different focal planes (i.e., different focusing positions). Areas within the corresponding focal plane are sharper, while out-of-focus areas are blurrier. To obtain a sharp image overall, it is necessary to identify the sharpest image at each pixel location within each focal plane image. This involves obtaining the results of accurate focusing at each location and then stitching the sharp parts together to obtain the final sharp image. Therefore, it is necessary to analyze the sharpness of each image to identify the sharp regions within each image.
[0064] For example, this application can achieve sharpness evaluation through grayscale gradient. For instance, the grayscale difference between each pixel in the focal plane image and its neighboring pixels can be calculated, and sharpness evaluation can be performed based on the grayscale difference. The larger the grayscale difference value, the sharper the pixel edge, the higher the sharpness, and the greater the sharpness weight.
[0065] Step 103: Perform image fusion based on the sharpness weight of each pixel in the focal plane image, and identify and locate the lesion area in the target area based on the generated fused image.
[0066] In one possible implementation, this application selects the pixel with the highest sharpness weight at the same pixel position in each focal plane image for image fusion to generate a fused image. Then, based on an image segmentation model, the lesion region in the target region is identified and located from the fused image.
[0067] For example, this application can iterate through each pixel position of the fused image, find the pixel at that pixel position in all focal plane images, compare the sharpness weights, and assign the grayscale value of the pixel with the largest sharpness weight to the corresponding pixel position of the fused image to generate the fused image.
[0068] After the fused image is generated, this application can locate the lesion region using an image segmentation model, such as a mask region-based convolutional neural network (Mask R-CNN). In this way, this application can place the lesion region in the center of the field of view by adjusting the microscope's field of view.
[0069] In some embodiments, the present application can execute the above technical solutions multiple times to achieve multiple positioning adjustments of the lesion area and generation of fused images, thereby further improving the image clarity and accuracy of the lesion area.
[0070] Based on the above technical solution, this application avoids the local blurring problem caused by the extremely small depth of field of a single focal plane by acquiring a sequence of microsurgical images under multiple focal planes. Then, by evaluating the sharpness of each pixel to determine the sharpness weight, it ensures that the sharpest pixel is used at each position in the fused image, effectively improving the overall sharpness of the full-focus image. Finally, the lesion area is located based on the clear fused image. Compared with the existing technical solutions, this application solves the problem of inaccurate positioning caused by local blurring in existing single focal plane images or simple stitched images, effectively improving the reliability and accuracy of lesion area positioning, and providing a reliable image data foundation for the intelligent and precise operation of intelligent multi-arm microsurgical robots.
[0071] As one possible embodiment of this application, in order to further improve the accuracy of the sharpness weight, combined with Figure 1 ,like Figure 2 As shown, step 102 above can be achieved through the following steps:
[0072] Step 201: For each focal plane image, determine the mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel.
[0073] The mean motion coefficient is used to characterize the overall tendency of motion blur in the focal plane image. The larger the mean motion coefficient, the higher the likelihood that the image as a whole is affected by motion blur. For example, this application can determine the motion coefficient of each partition by dividing the focal plane image (e.g., by using a clustering algorithm), thereby determining the mean motion coefficient of the entire focal plane image based on the motion coefficient of each partition.
[0074] The defocus evaluation coefficient is used to characterize the degree of blurring caused by defocusing of a pixel. The larger the defocus evaluation coefficient, the more severe the blurring caused by defocusing of the pixel.
[0075] It should be noted that microsurgery is not image acquisition of static objects. The structures in the target area (such as blood vessels) will move periodically with the patient's heartbeat and breathing, and the equipment itself will also vibrate slightly, causing image blurring. However, the above factors are motion blur caused by motion, not defocus blur caused by lens defocus. Existing technologies usually only consider image blur caused by lens defocus, and do not consider the impact of motion blur, thus making it difficult to ensure image sharpness. This application, however, can reasonably analyze the sharpness of each pixel by separately evaluating the impact of motion blur and defocus blur on image blur, thereby improving the fusion effect of subsequent image fusion.
[0076] Step 202: For each pixel in the focal plane image, determine the sharpness weight of the pixel based on the mean motion coefficient of the focal plane image, the defocus evaluation coefficient of each pixel, and the pixel position of each pixel.
[0077] It should be noted that, due to the physical characteristics of optical lenses, pixels closer to the center of the image usually have higher pixel clarity, and there may be field of view offset issues at the edges of the image. Therefore, when evaluating the clarity weight of a pixel, this application can also combine the pixel position of the pixel for evaluation.
[0078] For example, the sharpness weight of a pixel satisfies the following formula:
[0079]
[0080] in, For the first The first focal plane image Pixel sharpness weights For the first The first focal plane image The distance between each pixel and the center of the image on the focal plane. This represents the average distance between each pixel in the focal plane image and the center of the focal plane image. For example, the distance between a pixel and the center of the focal plane image can be determined by the pixel's position. For the first Mean motion coefficients of each focal plane image For the first The first focal plane image The defocus evaluation coefficient for each pixel. This represents the inverse proportional normalization function.
[0081] Based on the above technical solution, this application introduces the mean value of motion coefficients to characterize the overall motion blur tendency of the image, which can avoid misjudgment of sharpness caused by overall motion interference. By introducing a defocus evaluation coefficient to quantify defocus blur, it distinguishes it from blur caused by other factors. Simultaneously, by combining pixel position weights, it takes into account the potential field-of-view offset problem at image edges. In this way, this application can determine the sharpness weight through the above multi-dimensional parameters, making the calculation of the sharpness weight more closely match the actual interference scenarios of microsurgical images, further improving the accuracy of the sharpness weight, providing a more reliable basis for subsequent image fusion, and thus improving the detail restoration of the fused image.
[0082] Furthermore, since image sharpness primarily affects the display of transition regions between different components in an image, this application can determine the average motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel by analyzing the display of transition points in the transition regions of the focal plane image through screening transition points.
[0083] As one possible embodiment of this application, combined with Figure 2 ,like Figure 3 As shown, step 201 above can be achieved through the following steps:
[0084] Step 301: For each focal plane image, select multiple transition points from the focal plane image.
[0085] Transition points are pixels located at the edges of different tissues or structures in a focal plane image. These pixels have a significant difference in grayscale from their neighboring pixels, distinguishing them from pixels with uniform color within the tissue (such as pixels inside the same blood vessel, which have basically the same grayscale and are not considered transition points).
[0086] In one possible implementation, this application determines the uniformity of each pixel in each focal plane image based on the gray values of the pixel and its neighboring points. Then, pixels in the focal plane image whose uniformity is less than the uniformity threshold are selected as transition points.
[0087] Uniformity is used to characterize the degree of color uniformity in the neighborhood of a pixel. The higher the uniformity, the more likely the pixel is to be a uniform point within its composition; conversely, the lower the uniformity, the more likely it is to be a transition point. For example, grayscale values, also known as pixel values, are used to characterize the color of a pixel. They can be represented by three channels: red, green, and blue (RGB), with each channel corresponding to a color value.
[0088] For example, the uniformity of a pixel satisfies the following formula:
[0089]
[0090] in, The first in the focal plane image Uniformity of each pixel This represents the number of neighboring points of a pixel. For example, in the case of an 8-neighborhood, then... . This indicates the number of color channels. Taking RGB three channels as an example, then... . To indicate the first The first pixel The neighboring points at the th ... The value of each channel. Here The time refers to the first Each pixel itself. Indicates the first The pixel itself and its neighboring pixels in the th... The average value of each channel. This represents the inverse proportional normalization function.
[0091] For example, the uniformity threshold can be set according to the actual situation. For example, it can be set to 0.2. Pixels with a uniformity greater than or equal to 0.2 belong to the pixels inside the component, and pixels with a uniformity less than 0.2 belong to the transition points. That is, this application can filter pixels with large color changes as transition points of the component edge transition.
[0092] Step 302: Perform cluster analysis based on the diffusion direction of multiple transition points to obtain multiple clusters.
[0093] Each cluster includes one or more transition points. The diffusion direction is used to characterize the direction of gray-level change at the edge of the transition point. In this embodiment, the diffusion direction of the transition point can be determined by calculating the gray-level difference between the transition point and its neighboring points.
[0094] For example, cluster analysis can use the Affinity Propagation (AP) algorithm, which uses the diffusion direction of transition points as the clustering feature and performs clustering based on the similarity of diffusion directions (e.g., it can be represented by the angle between diffusion directions), ultimately obtaining multiple clusters. The diffusion directions of transition points within the clusters have strong consistency.
[0095] Step 303: Determine the mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel based on the diffusion direction of the transition points in multiple clusters.
[0096] It should be noted that because organs and tissues undergo periodic movements in response to a patient's heartbeat and respiration, slight movements of these organs and tissues during image acquisition can cause blurring in the images. This blurring could be due to defocusing caused by the focal length, or motion blur caused by the slight movements of the organs and tissues.
[0097] When the blur in an image is defocused blur, the direction of blur diffusion among pixels is not consistent. However, if the blur is caused by the micro-movement of an organ, the direction of blur will show a strong correlation with the direction of motion, and the blurred points will also show a consistent diffusion direction. Therefore, this application can evaluate the mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel by using the diffusion direction of transition points in multiple clusters.
[0098] Based on the above technical solution, this application focuses on the tissue edge pixels that have the greatest impact on image clarity by screening transition points, eliminating invalid interference from uniform points within the tissue, making subsequent blur analysis more targeted. By clustering the diffusion directions of transition points, transition points with consistent diffusion directions are grouped into a cluster, providing a data basis for distinguishing between motion blur and defocus blur (motion blur usually results in a high degree of consistency in the diffusion direction of a cluster of transition points, while defocus blur is characterized by irregular diffusion directions across multiple clusters). In this way, this application can determine the mean motion coefficient and defocus evaluation coefficient based on the clusters, providing a more reliable input for the accurate calculation of subsequent clarity weights, thereby improving the quality of the fused image and the accuracy of subsequent lesion area localization.
[0099] As one possible embodiment of this application, combined with Figure 3 ,like Figure 4 As shown, step 303 above can be achieved through the following steps:
[0100] Step 401: For each cluster, calculate the motion coefficient of the cluster based on the degree of spatial aggregation of transition points within the cluster and the consistency of diffusion direction.
[0101] In some embodiments, the degree of spatial clustering can be characterized by the average distance between transition points within a cluster. The smaller the average distance, the more spatially clustered the transition points are, and the more likely they are to be ambiguous regions of the same organization caused by movement. For example, this can be obtained by calculating the Euclidean distances between all pairwise transition points within a cluster and taking the average.
[0102] The consistency of diffusion direction can be characterized by the range of the angle between the diffusion directions of transition points within a cluster, that is, the difference between the maximum and minimum angles between the diffusion directions of a certain transition point and other transition points in the cluster; the smaller the range, the more consistent the diffusion directions of the transition points within the cluster.
[0103] For example, the motion coefficients of a cluster satisfy the following formula:
[0104]
[0105] in, For the first The motion coefficients of each cluster For the first The average distance between transition points within each cluster For the first The number of transition points in each cluster. For the first In the cluster, the th The range of the angles between the diffusion directions at each transition point can be determined by... calculate, For the first In the cluster, the th The maximum angle between the diffusion directions of each transition point and other transition points. For the first In the cluster, the th The minimum angle between the diffusion directions of a transition point and other transition points. This represents the inverse proportional normalization function.
[0106] Step 402: Obtain the mean motion coefficient of the focal plane image based on the motion coefficient of each cluster in the focal plane image.
[0107] For example, this application can calculate the arithmetic mean based on the motion coefficients of each cluster in the focal plane image, and use the calculation result as the mean of the motion coefficients of the focal plane image to characterize the overall tendency of motion blur in the focal plane image.
[0108] Step 403: Determine the defocus evaluation coefficient of each pixel based on the motion coefficient of each cluster in the focal plane image and the diffusion direction of the transition point.
[0109] It should be noted that the motion coefficient can characterize the tendency of motion blur at transition points within each cluster. Therefore, by combining the degree of blur based on the diffusion direction exhibited by each transition point itself, and by combining the motion coefficient to exclude its tendency of motion blur, it is possible to correct the degree of defocus, minimize the problem of overestimation of defocus due to the influence of motion blur, and thus accurately obtain the degree of defocus blur exhibited by each transition point.
[0110] For example, when the diffusion degree of the transition point itself is greater and the gray level change is more uniform, the blur degree exhibited by the transition point is greater. This application can combine the average motion coefficient of the focal plane image to make a certain correction to the diffusion direction of each transition point, thereby obtaining the defocus evaluation coefficient of each transition point.
[0111] Based on the above technical solution, this application evaluates the motion coefficient of clusters by combining spatial aggregation degree and directional consistency, which can more accurately identify the transition point clusters corresponding to motion blur, avoid misjudging defocus blur as motion blur, and determine the defocus evaluation coefficient by the correlation between motion coefficient and diffusion direction, effectively eliminating the interference of motion blur on defocus judgment, so that the defocus evaluation coefficient of each pixel is more in line with the actual defocus state, providing high-precision parameter support for the calculation of sharpness weight, and further improving the sharpness of fused image and the accuracy of lesion area localization.
[0112] As one possible embodiment of this application, this application can determine the diffusion direction of each transition point through gradient grayscale iterative analysis, combined with... Figure 4 ,like Figure 5As shown, the method also includes the following steps:
[0113] Step 501: For each transition point in the focal plane image, determine the gradient direction sequence based on the gradient gray-level change and gradient direction change of the transition point and its neighboring points.
[0114] The gradient direction sequence includes the transition point corresponding to each level of gradient.
[0115] The gradient grayscale change can be represented by the absolute value of the difference in grayscale values between adjacent pixels (such as a transition point and its neighboring points). It is used to characterize the degree of grayscale change at the edge of a pixel. The larger the gradient grayscale change, the sharper the edge.
[0116] The change in gradient direction can be represented by the angle between the gradient directions at the transition points of two adjacent gradient levels. This angle is used to characterize the continuity of the gradient direction. The smaller the change in gradient direction, the more consistent the gradient direction and the stronger the edge continuity.
[0117] For example, this application can determine the gradient direction sequence corresponding to the transition point through gradient gray-level iteration. For instance, taking a certain transition point in a focal plane image as an example, such as... Figure 6 and Figure 7 As shown, the maximum grayscale gradient between the transition point and the selected points (i.e., the neighboring points in the 8-neighborhood of the transition point) is determined as the first-level gradient. The direction from the transition point to the pixel corresponding to the first-level gradient is the first-level gradient direction. Then, the next level of gradient grayscale iteration begins. Centered on the pixel corresponding to the first-level gradient, the maximum grayscale gradient between the selected points and the pixel corresponding to the first-level gradient (i.e., the neighboring points in the 8-neighborhood of the pixel that are not in the previous iteration) is determined as the second-level gradient. The direction from the pixel to the pixel corresponding to the second-level gradient is the second-level gradient direction. This process continues in this manner. This method ensures that the deviation of the selected pixel from the previous level gradient direction is as small as possible, and the grayscale difference between the selected pixel and the corresponding point of the previous level gradient is as large as possible.
[0118] For example, this application can select the pixel corresponding to the current level gradient based on the gradient evaluation coefficient during gradient grayscale iteration. For instance, the gradient evaluation coefficient of a pixel satisfies the following formula:
[0119]
[0120] in, The first in the focal plane image The first transition point The first gradient in the first order Gradient evaluation coefficients of the candidate points For the first The first transition point The first gradient in the first order The first alternative point and the first The grayscale difference of pixels corresponding to the gradient can be obtained through... Calculations show that For the first The first transition point The first gradient in the first order The grayscale values of the candidate points For the first The first transition point The grayscale value of the pixel corresponding to the gradient level. For the first The first transition point The pixel corresponding to the first gradient and the second gradient The grayscale difference of pixels corresponding to the gradient level. For the first The transition point points to the first The first gradient in the first order The direction of the first alternative point is the same as the first... The transition point points to the first The angle between the direction of the pixel corresponding to the gradient is such that the smaller the angle, the more consistent the grayscale diffusion direction. Indicates the sign function, Indicates the first gray level difference and the first The value is 1 if the grayscale difference of each level is consistent, and -1 if they are inconsistent. For example, if the grayscale difference of the first level is consistent, the value is -1. gradient to gradient to When the gray value of the gradient is monotonically increasing or monotonically decreasing, the value is 1. Used to prevent the denominator from being equal to 0.
[0121] It should be noted that, for the gradient iteration process of the first-level gradient, the above... It can be represented as the first The first transition point The first alternative point and the first The grayscale difference of the pixels corresponding to each transition point It can be represented by a positive constant (such as 1). It can be 0.
[0122] For example, this application can determine whether to terminate the iteration based on the gradient evaluation coefficients of the aforementioned pixels. For instance, if there is a gradient evaluation coefficient greater than 0 among the gradient evaluation coefficients of the backup points obtained from the current level gradient, then the backup point with the largest gradient evaluation coefficient is taken as the current level gradient, and the next level gradient iteration is performed. If there is no gradient evaluation coefficient greater than 0 among the gradient evaluation coefficients of the backup points obtained from the current level gradient, that is, all gradient evaluation coefficients are less than 0, it indicates that the grayscale change in the next level candidate range does not conform to the previous pattern. At this time, the iteration is stopped, and a gradient direction sequence is generated based on the pixels corresponding to the gradients of each level obtained in the previous iteration process.
[0123] Step 502: Take the gradient direction between the transition point and the transition point corresponding to the last level gradient in the gradient direction sequence as the diffusion direction of the transition point.
[0124] In some embodiments, combined with Figure 4 ,like Figure 5 As shown, step 403 above can be achieved through the following steps:
[0125] Step 503: For each transition point in the focal plane image, take the last gradient level in the corresponding gradient direction sequence as the diffusion level of the transition point, and determine the diffusion evaluation coefficient of the transition point based on the gradient gray change of each level in the gradient direction sequence.
[0126] For example, the first If the gradient direction sequence of the k transition points includes k-level gradients, then this application can... The gradient of the first order is used as the gradient of the second order. The diffusion level of the nth transition point is determined by the average grayscale gradient between all adjacent gradient-corresponding pixels. The diffusion evaluation coefficient for each transition point. This diffusion evaluation coefficient is used to characterize the degree of drastic gray-level change along the gradient direction sequence at the transition point.
[0127] Step 504: Based on the motion coefficient of the cluster where the transition point is located, the diffusion direction of each transition point in the cluster where the transition point is located, and the diffusion level and diffusion evaluation coefficient of the transition point, determine the defocus evaluation coefficient of the transition point.
[0128] The transition point defocus evaluation coefficient is a quantitative value used to calculate the degree of defocus for transition points (tissue edge pixels). The larger the transition point defocus evaluation coefficient, the more severe the blur caused by defocus at the transition point.
[0129] For example, the defocus evaluation coefficient of the transition point satisfies the following formula:
[0130]
[0131] in, For the first The defocusing evaluation coefficient of each transition point For the first The motion coefficient of the cluster in which each transition point is located. For the first The diffusion direction of each transition point. For the first The direction of motion of a cluster containing a transition point can be represented by the average of the diffusion directions of each transition point in that cluster. This represents the cosine function. The consistency between the diffusion direction of the transition point and the motion direction of the cluster is used to characterize the degree of motion blur rather than defocus blur. For the first The diffusion level at each transition point For the first Diffusion evaluation coefficient for each transition point. This is the normalization function.
[0132] Step 505: Divide the focal plane image into multiple connected regions based on the connectivity between each transition point in the focal plane image.
[0133] Each connected region comprises multiple pixels. A connected region is a region with uniform pixel grayscale (such as inside a blood vessel or a normal brain tissue region) surrounded by multiple transition points (tissue edges), and the defocus state of pixels within the region is consistent with that of the surrounding transition points. Therefore, this application can divide a focal plane image into multiple connected regions according to connectivity, thereby evaluating the defocus evaluation coefficient of each pixel based on the connected regions.
[0134] Step 506: For each connected component, take the average value of the defocus evaluation coefficient of each transition point in the connected component as the defocus evaluation coefficient of each pixel in the connected component.
[0135] Based on the above technical solution, this application achieves effective separation of motion blur and defocus blur by defining a defocus evaluation coefficient for transition points, avoiding evaluation bias caused by conflating the two types of blur. At the same time, a connected component mean strategy is adopted for non-transition points, which not only ensures the consistency of defocus evaluation of pixels within the domain, but also reduces the computational power consumption of pixel-by-pixel calculation, balancing accuracy and efficiency. This provides highly accurate defocus quantification parameters for subsequent sharpness weight calculation, further improving the sharpness of the fused image and the reliability of lesion area localization.
[0136] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0137] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
Claims
1. A method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot, characterized in that, The lesion area visual localization method for intelligent multi-arm microsurgical robots includes: A sequence of microsurgical images of the target region under multiple focal planes is acquired; the sequence of microsurgical images includes focal plane images corresponding to the multiple focal planes; the pixels in the focal plane images corresponding to the multiple focal planes are in one-to-one correspondence. For each focal plane image, the sharpness of each pixel in the focal plane image is evaluated, and the sharpness weight of each pixel is determined. Image fusion is performed based on the sharpness weights of pixels in each focal plane image, and the lesion area in the target region is identified and located based on the generated fused image; For each focal plane image, the sharpness of each pixel in the focal plane image is evaluated to determine the sharpness weight of each pixel, including: For each focal plane image, the mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel are determined; the mean motion coefficient is used to characterize the overall tendency of motion blur in the focal plane image; the defocus evaluation coefficient is used to characterize the degree of blurring caused by defocusing of the pixel. For each pixel in the focal plane image, the sharpness weight of the pixel is determined based on the mean motion coefficient of the focal plane image, the defocus evaluation coefficient of each pixel, and the pixel position of each pixel. For each focal plane image, determining the mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel includes: For each focal plane image, multiple transition points are selected from the focal plane image; the transition points are pixels located in the edge regions of different tissues or structures in the focal plane image. Cluster analysis is performed based on the diffusion direction of the multiple transition points to obtain multiple clusters; each cluster includes one or more transition points. The mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel are determined based on the diffusion direction of the transition points in the multiple clusters. The determination of the mean motion coefficient of the focal plane image and the defocus evaluation coefficient of each pixel based on the diffusion direction of the transition points in the multiple clusters includes: For each cluster, the motion coefficient of the cluster is calculated based on the degree of spatial aggregation of transition points within the cluster and the consistency of diffusion direction. The mean motion coefficient of the focal plane image is obtained based on the motion coefficient of each cluster in the focal plane image. The defocus evaluation coefficient of each pixel is determined based on the motion coefficient of each cluster in the focal plane image and the diffusion direction of the transition point.
2. The method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot according to claim 1, characterized in that, For each focal plane image, multiple transition points are selected from the focal plane image, including: For each pixel in each focal plane image, the uniformity of the pixel is determined based on the gray values of the pixel and its neighboring points. Pixels in the focal plane image whose uniformity is less than the uniformity threshold are selected as transition points.
3. The method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot according to claim 1, characterized in that, The lesion area visual localization method for intelligent multi-arm microsurgical robots also includes: For each transition point in the focal plane image, a gradient direction sequence is determined based on the gradient grayscale change and gradient direction change of the transition point and its neighboring points; the gradient direction sequence includes the transition points corresponding to each level of gradient of the transition point. The gradient direction between the transition point and the transition point corresponding to the last level gradient in the gradient direction sequence is taken as the diffusion direction of the transition point.
4. The lesion area visual localization method for intelligent multi-arm microsurgical robots according to claim 3, characterized in that, The process of determining the defocus evaluation coefficient for each pixel based on the motion coefficient of each cluster in the focal plane image and the diffusion direction of the transition points includes: For each transition point in the focal plane image, the last gradient level in the corresponding gradient direction sequence is taken as the diffusion level of the transition point, and the diffusion evaluation coefficient of the transition point is determined based on the gradient gray-scale change of each level in the gradient direction sequence. Based on the motion coefficient of the cluster in which the transition point is located, the diffusion direction of each transition point in the cluster in which the transition point is located, and the diffusion level and diffusion evaluation coefficient of the transition point, the defocus evaluation coefficient of the transition point is determined. Based on the connectivity between each transition point in the focal plane image, the focal plane image is divided into multiple connected regions; each connected region includes multiple pixels. For each connected component, the average value of the defocus evaluation coefficient of each transition point in the connected component is used as the defocus evaluation coefficient of each pixel in the connected component.
5. The method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot according to claim 1, characterized in that, The sequence of microsurgical images of the target area under multiple focal planes includes: Based on the depth range of the target area, determine the start point, end point, and focal plane step size of the acquisition path; The electric focusing device controlling the intelligent multi-arm microsurgical robot moves step by step from the starting point to the ending point according to the focal plane step size, and triggers camera exposure at each focal plane position to capture the corresponding focal plane image.
6. The method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot according to claim 5, characterized in that, The captured focal plane image is triggered based on real-time monitoring of the target object's physiological signals, which are used to indicate whether the target object is in a quiescent state.
7. The method for visual localization of lesion areas in an intelligent multi-arm microsurgical robot according to claim 1, characterized in that, The process of image fusion based on the sharpness weights of pixels in each focal plane image, and identifying and locating lesion areas in the target region based on the generated fused image, includes: For pixels at the same pixel location in each focal plane image, the pixel with the highest sharpness weight at that pixel location is selected for image fusion to generate the fused image; Based on an image segmentation model, the lesion region in the target region is identified and located from the fused image.