Light focusing control method and system in operation process

Abstract

The invention discloses a light focusing control method and system in an operation process, and the method comprises the steps: obtaining a corresponding dynamic image in an operating room to obtain a pixel track of a mechanical arm, and controlling an operating lamp to move to a target focusing position according to the pixel track; obtaining a face image and an eye image of an operation implementation person when the operation lamp is located at the target focusing position, and inputting the eye image and the face image into a pre-trained deep learning model to obtain a sight angle of the operation implementation person, obtaining a target irradiation angle of the operating lamp according to the sight angle; and acquiring a light spot image corresponding to a surgical operation area when the surgical lamp performs irradiation according to the target irradiation angle, and performing size analysis on the light spot image so as to adjust the irradiation focal length of the surgical lamp according to a size analysis result, thereby improving the efficiency and accuracy of light focusing control.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a light focusing control method and system during a surgery. Background Art

[0002] Surgery has extremely high requirements for operating precision, and the light from surgical lamps plays a vital role in ensuring doctors have a clear view. Existing surgical lamps are all ceiling-mounted or floor-standing, and their illumination angles cannot be adjusted adaptively. They often rely on third-party operators, which not only causes delays in light movement, but also the operator's understanding of the specific illumination effect required by the doctor may be inaccurate, thus affecting the efficiency and effect of the operation.

[0003] Furthermore, during the adjustment process of the operation, since various types of operations target different surgical sites, medical staff need to adjust the irradiation direction or irradiation distance of the lamp head when adjusting the position of the operating light. This adjustment process depends on the continuous attempts of medical staff, which not only fails to meet the requirements of the operation in a timely manner, but also may cause changes in optical parameters during the adjustment process, making the state of the optical parameters unstable, thereby affecting the effect of the operation.

[0004] In recent years, with the development of science and technology and the requirements for standardization of surgical procedures, robotic arms have gradually been applied to the surgical process to perform more precise surgical operations or detect surgical progress, thereby providing more accurate data basis for subsequent surgical review. Summary of the invention

[0005] In order to solve the above technical problems, the present invention discloses a light focus control method and system during a surgery, which are used to improve the efficiency and accuracy of light focus control.

[0006] In order to achieve the above object, the present invention discloses a method for controlling light focus during surgery, comprising: Acquire a corresponding dynamic image in the operating room, and acquire a pixel trajectory of the robotic arm when it moves in the dynamic image; Performing coordinate system conversion on the pixel trajectory to obtain trajectory parameters corresponding to the movement of the robot arm and controlling the surgical light to move to a target focusing position according to the trajectory parameters; Acquire a facial image of a person performing surgery when the surgical light is located at the target focus position, and extract eye key points from the facial image to extract a corresponding eye image from the facial image; Inputting the eye image and the facial image into a pre-trained deep learning model, so as to obtain the sight angle of the operator through the deep learning model, and then obtaining the target irradiation angle of the surgical lamp according to the sight angle; Acquire a light spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform a size analysis on the light spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis; The surgical lamp is controlled to perform focused illumination on the surgical operation area according to the adjusted illumination focal length.

[0007] The present invention discloses a method for controlling light focus in a surgical process, which first obtains a corresponding dynamic image in the operating room, extracts the moving trajectory of a robotic arm used to track the surgical process from the dynamic image, and then controls the surgical light to move according to the moving trajectory. Among them, since the robotic arm always tracks the surgical process of the operator, when the position of the surgical light is adjusted according to the moving trajectory of the robotic arm, it can be ensured that the illumination of the surgical light fits the surgical requirements of the operator without manual adjustment, thereby ensuring both the efficiency and accuracy of the surgical light adjustment.

[0008] Furthermore, after adjusting the position of the surgical light itself according to the movement of the robotic arm, the irradiation target of the surgical light also changes due to the movement of the position. In order to ensure that the irradiation direction of the surgical light is consistent with the irradiation requirements of the surgeon, the present application obtains the surgeon's line of sight to adjust the irradiation angle of the surgical light, thereby improving the accuracy of the irradiation of the surgical light. Furthermore, after adjusting the irradiation angle, in order to avoid the irradiation of the surgical light affecting the surgical field of view of the surgeon, the corresponding spot image in the current irradiation direction can be obtained, and the irradiation distance of the surgical light can be adjusted by performing a size analysis on the spot image, thereby avoiding overexposure that affects the surgical process and ensuring the effect of the adjustment.

[0009] As a preferred example, the obtaining of the corresponding dynamic image in the operating room includes: Splitting the dynamic image into a plurality of continuous static image frames; Performing grayscale processing on each of the static image frames to extract the outline of the robotic arm in the static image frame; wherein the robotic arm is used to track the progress of the surgery; Filtering a corresponding robotic arm region in the static image frame according to the contour to extract pixel feature information corresponding to the robotic arm region; The static image frame is divided into a plurality of sub-mechanical arm regions by using a clustering algorithm, and the sub-mechanical arm regions whose pixel feature information ratios with the mechanical arm region are within a predetermined range are merged to obtain a plurality of continuous mechanical arm image frames.

[0010] After acquiring the corresponding dynamic image in the operating room, the present invention divides each static image frame in the dynamic image into regions to avoid interference of other backgrounds in the image with the recognition of the movement of the robotic arm, thereby improving the accuracy of the recognition of the movement of the robotic arm. At the same time, by filtering out other background pixels in the image, the amount of subsequent data processing is reduced, thereby improving the efficiency of trajectory recognition, thereby improving the efficiency and accuracy of lighting control.

[0011] As a preferred example, the step of obtaining a pixel trajectory of the robot arm when it moves in the dynamic image includes: Based on a plurality of the robotic arm image frames, an image matrix difference calculation is performed to obtain a difference between two preceding and following robotic arm image frames to generate a difference image; Obtaining the contour of the difference image, and saving the contour and its centroid and coordinate values; The pixel trajectory is obtained based on the centroids of the contours of all the difference images.

[0012] The present invention obtains the contour of the difference image by comparing the difference between consecutive frames, so that the center of mass of the moving trajectory can be obtained through the contour, and then the movement of the center of mass is used to describe the moving trajectory of the robotic arm, thereby ensuring the accuracy of trajectory recognition.

[0013] As a preferred example, the coordinate system conversion of the pixel trajectory to obtain trajectory parameters corresponding to the movement of the robot arm and controlling the surgical light to move to the target focus position according to the trajectory parameters includes: Converting the coordinate system of the pixel trajectory into a world coordinate system to obtain the world coordinate system movement trajectory of the robotic arm; By using moveit path planning, trajectory points including time information, acceleration information and speed information are obtained according to the movement trajectory of the world coordinate system, and spline interpolation is performed on the trajectory points for a preset number of times to obtain trajectory parameters when the robot arm moves; The surgical light is controlled to move according to the trajectory parameters according to the movement trajectory of the world coordinate system, so as to reproduce the movement trajectory of the robot arm in the dynamic image until it moves to the target focusing position.

[0014] After the present invention generates the movement trajectory of the robot arm in the image through the change of pixels in the image, in order to achieve the adjustment of the surgical light, it is necessary to convert the movement in the image into movement in the real world. After moving the coordinate system, path planning is used to generate trajectory parameters of the movement trajectory to ensure the accuracy of the adjustment of the surgical light based on the trajectory parameters.

[0015] As a preferred example, the step of extracting eye key points from the facial image to extract a corresponding eye image from the facial image includes: Processing the facial image by using a pre-trained facial detection model and a facial key point detection model respectively to obtain a facial detection frame and facial key points; The face is cropped according to the face detection frame to obtain face image data, and the face image is cropped according to the face key points to obtain the eye image; wherein the eye image includes a left eye image and a right eye image.

[0016] When estimating the line of sight angle of a surgeon, the present invention simultaneously obtains an eye image and a facial image of the surgeon to ensure the accuracy of the estimation, thereby performing synchronous posture estimation on the facial image and the eye image, and improving the accuracy of the line of sight estimation based on the posture correction between the face and the eyes, thereby improving the accuracy of the adjustment of the irradiation angle of the surgical light.

[0017] As a preferred example, the step of obtaining the sight angle of the operator performing the surgery through the deep learning model includes: Performing convolution processing on the facial image and the eye image respectively through the convolution network preset in the deep learning model to extract a first posture feature corresponding to the facial image and a second posture feature corresponding to the eye image; Obtaining the sight line feature corresponding to the surgical operator by performing a weighted summation of the first posture feature and the second posture feature through a fully connected layer preset in the deep learning model; The sight line feature is input into a classifier preset in the deep learning model, and the sight line angle corresponding to the surgical personnel is output through the classifier.

[0018] The present invention utilizes the deep learning model to fuse the facial and eye posture features to perform line of sight estimation, thereby improving the accuracy of line of sight estimation. Furthermore, a fully connected layer is set in the deep learning model to perform weighted summation of the features, thereby reducing the amount of data processing and thereby reducing the training cost of the model.

[0019] As a preferred example, obtaining the target irradiation angle of the surgical lamp according to the sight angle includes: Acquiring the head coordinate position of the operator to calculate the focus coordinates of the operator's line of sight focus point according to the head coordinate position and the line of sight angle; The focusing coordinates and the current target focusing position of the surgical light are substituted into a preset focusing angle calculation formula to obtain a target irradiation angle corresponding to the surgical light.

[0020] The present invention utilizes the estimated sight angle and combines it with the head position of the surgeon to obtain the surgeon's corresponding sight focus, and then determines the illumination angle of the surgical light according to the sight focus and the current position information of the surgical light, thereby ensuring that the illumination angle of the surgical light meets the focusing requirements of the surgeon and improving the accuracy of the surgical light adjustment.

[0021] As a preferred example, the performing size analysis on the light spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis includes: Calculating a standard moment corresponding to the spot image, so as to calculate an image center of the spot image according to the standard moment; Obtaining a fitted spot center point according to the image center and a preset air dynamic curve fitting algorithm, and then calculating the size of the spot image according to the spot center point; A spot compensation value is determined according to the size and a preset size threshold, and then the irradiation focal length of the surgical lamp is adjusted according to the spot compensation value.

[0022] After adjusting the angle and position, in order to avoid overexposure and thus affect the surgical field of the surgeon, the present invention analyzes the size of the corresponding spot image under the current illumination and adjusts the illumination distance of the surgical lamp according to the size, thereby ensuring that the surgical lamp provides the surgeon with a good surgical field of view at the illumination distance and improves the focusing effect.

[0023] On the other hand, the present invention discloses a light focusing control system in a surgical process, including a trajectory tracking module, a coordinate conversion module, an image acquisition module, a sight estimation module, an angle adjustment module and a distance adjustment module; The trajectory tracking module is used to obtain the corresponding dynamic image in the operating room, and obtain the pixel trajectory of the robot arm when it moves in the dynamic image; The coordinate conversion module is used to perform coordinate system conversion on the pixel trajectory to obtain trajectory parameters corresponding to the movement of the robot arm and control the surgical light to move to the target focus position according to the trajectory parameters; The image acquisition module is used to obtain a facial image of a surgical practitioner when the surgical light is located at the target focus position, and to extract eye key points from the facial image to extract a corresponding eye image from the facial image; The sight line estimation module is used to input the eye image and the face image into a pre-trained deep learning model, so as to obtain the sight line angle of the operator through the deep learning model, and then obtain the target irradiation angle of the surgical lamp according to the sight line angle; The angle adjustment module is used to obtain a light spot image corresponding to the surgical operation area when the surgical lamp is irradiated according to the target irradiation angle, and to perform a size analysis on the light spot image, so as to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis; The distance adjustment module is used to control the surgical lamp to perform focused illumination on the surgical operation area according to the adjusted illumination focal length.

[0024] The present invention discloses a light focusing control system in a surgical process, which first obtains a corresponding dynamic image in the operating room, extracts the moving trajectory of a mechanical arm used to track the surgical process from the dynamic image, and then controls the surgical light to move according to the moving trajectory. Among them, since the mechanical arm always tracks the surgical process of the operator, when the position of the surgical light is adjusted according to the moving trajectory of the mechanical arm, it can be ensured that the illumination of the surgical light fits the surgical requirements of the operator without manual adjustment, thereby ensuring both the efficiency and accuracy of the surgical light adjustment.

[0025] Furthermore, after adjusting the position of the surgical light itself according to the movement of the robotic arm, the irradiation target of the surgical light also changes due to the movement of the position. In order to ensure that the irradiation direction of the surgical light is consistent with the irradiation requirements of the surgeon, the present application obtains the surgeon's line of sight to adjust the irradiation angle of the surgical light, thereby improving the accuracy of the irradiation of the surgical light. Furthermore, after adjusting the irradiation angle, in order to avoid the irradiation of the surgical light affecting the surgical field of view of the surgeon, the corresponding spot image in the current irradiation direction can be obtained, and the irradiation distance of the surgical light can be adjusted by performing a size analysis on the spot image, thereby avoiding overexposure that affects the surgical process and ensuring the effect of the adjustment.

[0026] As a preferred example, the trajectory tracking module includes an image parsing unit and a trajectory building unit; The image analysis unit is used to split the dynamic image into a number of continuous static image frames; perform grayscale processing on each of the static image frames to extract the contour of the robotic arm in the static image frame; wherein the robotic arm is used to track the progress of the surgery; screen the corresponding robotic arm region in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm region; use a clustering algorithm to divide the static image frame into a plurality of sub-robotic arm regions, and merge the sub-robotic arm regions whose pixel feature information ratios with the robotic arm region are within a predetermined range to obtain a number of continuous robotic arm image frames; The trajectory building unit is used to perform image matrix difference calculation based on several of the robotic arm image frames to obtain the difference between the two preceding and succeeding robotic arm image frames to generate a difference image; obtain the contour of the difference image, save the contour and its center of mass and coordinate values; and obtain the pixel trajectory based on the center of mass of the contours of all the difference images.

[0027] After acquiring the corresponding dynamic image in the operating room, the present invention divides each static image frame in the dynamic image into regions to avoid interference of other backgrounds in the image with the recognition of the movement of the robotic arm, thereby improving the accuracy of the recognition of the movement of the robotic arm. At the same time, by filtering out other background pixels in the image, the amount of subsequent data processing is reduced, thereby improving the efficiency of trajectory recognition, thereby improving the efficiency and accuracy of lighting control.

[0028] Furthermore, by comparing the differences between consecutive frames, the contour of the difference image is obtained, so that the center of mass of the moving trajectory can be obtained through the contour, and then the movement of the center of mass is used to describe the moving trajectory of the robotic arm, thereby ensuring the accuracy of trajectory recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 : is a flow chart of a method for controlling light focusing during a surgery disclosed in an embodiment of the present invention; Figure 2 : is a schematic diagram of the structure of a light focusing control system in a surgical procedure disclosed in an embodiment of the present invention; Figure 3 : A flow chart of a method for controlling light focusing during a surgical procedure disclosed in another embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Embodiment 1 This embodiment discloses a method for controlling light focus during surgery. Specifically, the specific implementation process of the control method can be referred to Figure 1 , mainly including steps 101 to 106, the steps mainly include: Step 101: Acquire a corresponding dynamic image in the operating room, and acquire a pixel trajectory of the robot arm when it moves in the dynamic image.

[0032] In this embodiment, this step mainly includes: splitting the dynamic image into several consecutive static image frames; performing grayscale processing on each of the static image frames to extract the outline of the robotic arm in the static image frame; wherein the robotic arm is used to track the progress of the operation; screening the corresponding robotic arm area in the static image frame according to the outline to extract the pixel feature information corresponding to the robotic arm area; using a clustering algorithm to divide the static image frame into multiple sub-robotic arm areas, and merging the sub-robotic arm areas whose pixel feature information ratio with the robotic arm area is within a predetermined range to obtain several consecutive robotic arm image frames.

[0033] Furthermore, based on a number of the robotic arm image frames, an image matrix difference calculation is performed to obtain the difference between the two preceding and succeeding robotic arm image frames to generate a difference image; the contour of the difference image is obtained, and the contour and its center of mass and coordinate values ​​are saved; based on the center of mass of the contours of all the difference images, the pixel trajectory is obtained.

[0034] In this embodiment, after acquiring the corresponding dynamic image in the operating room, this step divides each static image frame in the dynamic image into regions to avoid interference from other backgrounds in the image in identifying the movement of the robotic arm, thereby improving the accuracy of identifying the movement of the robotic arm. At the same time, by filtering out other background pixels in the image, the amount of subsequent data processing is reduced, thereby improving the efficiency of trajectory recognition, thereby improving the efficiency and accuracy of lighting control. Furthermore, by comparing the difference between consecutive frames, the contour of the difference image is obtained, so that the center of mass of the movement trajectory can be obtained through the contour, and then the movement of the center of mass is used to describe the movement trajectory of the robotic arm, thereby ensuring the accuracy of trajectory recognition.

[0035] Step 102: performing coordinate system conversion on the pixel trajectory to obtain trajectory parameters corresponding to the movement of the robot arm and controlling the surgical light to move to a target focus position according to the trajectory parameters.

[0036] In this embodiment, this step mainly includes: converting the coordinate system of the pixel trajectory into a world coordinate system to obtain the world coordinate system movement trajectory of the robotic arm; obtaining trajectory points containing time information, acceleration information and speed information according to the world coordinate system movement trajectory through moveit path planning, and performing spline interpolation on the trajectory points for a preset number of times to obtain trajectory parameters when the robotic arm moves; controlling the surgical light to move according to the world coordinate system movement trajectory according to the trajectory parameters, so as to execute the reproduction of the movement trajectory of the robotic arm in the dynamic image until it moves to the target focus position.

[0037] In this embodiment, after generating the movement trajectory of the robot arm in the image through the changes in pixels in the image, in order to achieve the adjustment of the surgical light, it is necessary to convert the movement in the image into movement in the real world. After moving the coordinate system, path planning is used to generate trajectory parameters of the movement trajectory to ensure the accuracy of the adjustment of the surgical light based on the trajectory parameters.

[0038] Step 103: Acquire a facial image of the operator when the surgical light is located at the target focus position, and extract eye key points from the facial image to extract a corresponding eye image from the facial image.

[0039] In this embodiment, this step mainly includes: processing the facial image through a pre-trained face detection model and a facial key point detection model respectively to obtain a facial detection frame and facial key points; cropping the face according to the facial detection frame to obtain facial image data and cropping the facial image through the facial key points to obtain the eye image; wherein the eye image includes a left eye image and a right eye image.

[0040] In this embodiment, when estimating the line of sight angle of the surgeon, in order to ensure the accuracy of the estimation, this step simultaneously obtains the eye image and the face image of the surgeon, and synchronously estimates the posture of the facial image and the eye image, and improves the accuracy of the line of sight estimation based on the posture correction between the face and the eyes, thereby improving the accuracy of the adjustment of the illumination angle of the surgical light.

[0041] Step 104: Input the eye image and the facial image into a pre-trained deep learning model to obtain the sight angle of the surgeon through the deep learning model, and then obtain the target illumination angle of the surgical light according to the sight angle.

[0042] In this embodiment, this step mainly includes: performing convolution processing on the facial image and the eye image respectively through the convolution network preset in the deep learning model, and extracting the first posture feature corresponding to the facial image and the second posture feature corresponding to the eye image; performing weighted summation on the first posture feature and the second posture feature through the fully connected layer preset in the deep learning model to obtain the line of sight feature corresponding to the surgical performer; inputting the line of sight feature into the classifier preset in the deep learning model, and outputting the line of sight angle corresponding to the surgical performer through the classifier.

[0043] Furthermore, the head coordinate position of the surgeon is obtained to calculate the focus coordinates of the surgeon's line of sight focus point based on the head coordinate position and the line of sight angle; the focus coordinates and the current target focus position of the surgical light are substituted into a preset focus angle calculation formula to obtain the target irradiation angle corresponding to the surgical light.

[0044] In this embodiment, this step utilizes the deep learning model to fuse the facial and eye posture features to perform line of sight estimation, thereby improving the accuracy of line of sight estimation. Furthermore, a fully connected layer is set in the deep learning model to perform weighted summation on the features, thereby reducing the amount of data processing and thereby reducing the model training cost. Furthermore, the estimated line of sight angle is used in combination with the head position of the surgeon to obtain the corresponding line of sight focus of the surgeon, and then the illumination angle of the surgical light is determined based on the line of sight focus and the current position information of the surgical light, thereby ensuring that the illumination angle of the surgical light meets the focusing requirements of the surgeon and improving the accuracy of surgical light adjustment.

[0045] Step 105: Acquire a light spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform size analysis on the light spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis.

[0046] In this embodiment, this step mainly includes: calculating the standard moment corresponding to the spot image to calculate the image center of the spot image according to the standard moment; obtaining the fitted spot center point according to the image center and a preset dynamic curve fitting algorithm for the air, and then calculating the size of the spot image according to the spot center point; determining the spot compensation value according to the size and a preset size threshold, and then adjusting the irradiation focal length of the surgical lamp according to the spot compensation value.

[0047] In this embodiment, after adjusting the angle and position, in order to avoid overexposure and thus affecting the surgical field of view of the surgeon, the size of the corresponding spot image under the current illumination is analyzed to adjust the illumination distance of the surgical lamp according to the size, thereby ensuring that the surgical lamp provides the surgeon with a good surgical field of view at the illumination distance and improving the focusing effect.

[0048] Step 106: Control the surgical lamp to perform focused illumination on the surgical operation area according to the adjusted illumination focal length.

[0049] On the other hand, this embodiment also discloses a light focusing control system during surgery. For the specific structure of the control system, please refer to Figure 2 , including a trajectory tracking module 201, a coordinate conversion module 202, an image acquisition module 203, a sight estimation module 204, an angle adjustment module 205 and a distance adjustment module 206.

[0050] The trajectory tracking module 201 is used to obtain the corresponding dynamic image in the operating room, and obtain the pixel trajectory of the robot arm when it moves in the dynamic image.

[0051] The coordinate conversion module 202 is used to perform coordinate system conversion on the pixel trajectory to obtain trajectory parameters corresponding to the movement of the robot arm and control the surgical light to move to the target focus position according to the trajectory parameters.

[0052] The image acquisition module 203 is used to obtain the facial image of the operator when the surgical light is located at the target focus position, and to extract eye key points from the facial image to extract the corresponding eye image from the facial image.

[0053] The sight line estimation module 204 is used to input the eye image and the facial image into a pre-trained deep learning model to obtain the sight line angle of the surgeon through the deep learning model, and then obtain the target illumination angle of the surgical light based on the sight line angle.

[0054] The angle adjustment module 205 is used to obtain the light spot image corresponding to the surgical operation area when the surgical light is irradiated according to the target irradiation angle, and perform size analysis on the light spot image to adjust the irradiation focal length of the surgical light according to the result of the size analysis.

[0055] The distance adjustment module 206 is used to control the surgical lamp to perform focused illumination on the surgical operation area according to the adjusted illumination focal length.

[0056] In this embodiment, the trajectory tracking module 201 includes an image parsing unit and a trajectory building unit.

[0057] The image analysis unit is used to split the dynamic image into several consecutive static image frames; perform grayscale processing on each of the static image frames to extract the outline of the robotic arm in the static image frame; wherein the robotic arm is used to track the progress of the operation; filter the corresponding robotic arm area in the static image frame according to the outline to extract the pixel feature information corresponding to the robotic arm area; use a clustering algorithm to divide the static image frame into multiple sub-robotic arm areas, and merge the sub-robotic arm areas whose pixel feature information ratio with the robotic arm area is within a predetermined range to obtain several consecutive robotic arm image frames.

[0058] The trajectory building unit is used to perform image matrix difference calculation based on several of the robotic arm image frames to obtain the difference between the two preceding and succeeding robotic arm image frames to generate a difference image; obtain the contour of the difference image, save the contour and its center of mass and coordinate values; and obtain the pixel trajectory based on the center of mass of the contours of all the difference images.

[0059] Embodiment 2 In traditional surgical procedures, the position and angle of lights are manually adjusted to provide the surgeon with a surgical field of view. However, when encountering special circumstances, the technology of manually adjusting lights cannot effectively adjust the angle and distance of light focus in a timely manner, which in turn affects the surgical process.

[0060] In this regard, in certain implementations of this embodiment, the detection of the medical robotic arm during the operation is used as a reference point for adjusting the light, and the movement of the light is controlled by monitoring the movement vector of the medical robotic arm, thereby achieving effective real-time light adjustment.

[0061] Specifically, refer to Figure 3 , is a flow chart of a method for controlling light focus in a surgical procedure provided by this embodiment, which mainly includes steps 301 to 306, wherein the steps are mainly: Step 301: Obtain a surveillance video in an operating room, and perform instance segmentation on the surveillance video to obtain a dynamic image that only includes a medical robotic arm.

[0062] In this embodiment, the steps mainly include: splitting the dynamic image into several consecutive static image frames; performing grayscale processing on each of the static image frames to extract the outline of the robotic arm in the static image frame; wherein the robotic arm is used to track the progress of the operation; screening the corresponding robotic arm area in the static image frame according to the outline to extract the pixel feature information corresponding to the robotic arm area; using a clustering algorithm to divide the static image frame into multiple sub-robotic arm areas, and merging the sub-robotic arm areas whose pixel feature information ratio with the robotic arm area is within a predetermined range to obtain several consecutive robotic arm image frames.

[0063] Specifically, in some implementations of this embodiment, the real-time surveillance video is first split into continuous image frames along the time axis, and then each of the image frames is gray-scaled to obtain the contour of the robotic arm. The contour can be extracted using the classic Canny algorithm or the open source OpenCV model.

[0064] Furthermore, after extracting the contour, the area where the robotic arm is located is cut out from the image frame along the contour, and then pixel feature information of the area where the robotic arm is located is counted, so as to further screen the image frame according to the pixel feature information to improve the accuracy of robotic arm extraction.

[0065] Specifically, in the process of calculating the pixel feature information, the pixel points of the area where the robotic arm is located are first counted, and then the center position of the area is calculated to obtain the hue, saturation and intensity of the pixels in the center position, and then obtain the color characteristics of the area.

[0066] In some implementations of this embodiment, the calculation formula for counting the pixels is as follows:

[0067] Wherein, S represents the total number of pixels that meet the conditions, W represents the length of the region, H represents the width of the region, Represents a pixel at any point in the area, and the pixel is not a white pixel.

[0068] All pixel points in the area are counted according to the calculation formula to obtain the total number of pixels in the area, and then the pixel mean in the area is calculated based on the total number of pixels.

[0069] Furthermore, when calculating the center position of the area, the location information corresponding to the center area is calculated using a preset center area calculation formula. The center area calculation formula is expressed as:

[0070] Among them, the represents the coordinates of the upper left corner corresponding to the central area, Indicates the coordinates of the lower right corner corresponding to the central area.

[0071] Next, the hue, saturation and intensity of the pixels corresponding to the central area are calculated. Specifically, the calculation formulas for the hue, saturation and intensity are as follows:

[0072] Among them, the represents the average value of chromaticity C in the central area; represents the average value of the intensity I in the central area; represents the average value of the saturation SAT in the central area; and S represents the total number of pixels that meet the conditions.

[0073] After acquiring the hue, saturation and intensity of the pixels in the center position and obtaining the color features of the area, a clustering algorithm is used to divide each of the image frames into multiple sub-images to obtain multiple sub-areas. After obtaining the color features corresponding to each sub-area using the same color feature extraction algorithm, the sub-robotic arm areas whose pixel feature information ratios with the center area are within a predetermined range are merged to obtain a number of continuous robotic arm image frames.

[0074] Step 302: Obtain the pixel movement trajectory of the robot arm when it moves in the dynamic image, and perform coordinate system conversion on the pixel movement trajectory to generate the world coordinate system movement trajectory of the surgical light.

[0075] In this embodiment, this step mainly includes: performing image matrix difference calculation based on several of the robotic arm image frames, obtaining the difference between the two preceding and succeeding robotic arm image frames to generate a difference picture, obtaining the contour of the difference picture, saving the contour and its center of mass and coordinate values, and obtaining the pixel trajectory based on the center of mass of the contours of all the difference pictures.

[0076] Specifically, in certain implementations of this embodiment, an image matrix difference calculation is performed based on a number of static image frames to obtain the difference between the two static image frames before and after, and a new difference image is generated. Based on the centroid of the contours of all difference images, a pixel coordinate system trajectory is obtained. The pixel coordinate system trajectory is then converted to a world coordinate system trajectory. Specifically, the pixel coordinate system is first converted to a camera coordinate system, and then the camera coordinate system is converted to a world coordinate system according to the calibration process of the camera. The relationship between the pixel coordinate system and the camera coordinate system is as follows:

[0077] in, are the coordinates of the centroid in the pixel coordinate system, is the camera coordinate system, is the pixel coordinate system of the center point of the image, In the camera intrinsics, it is expressed as the width of the unit pixel.

[0078] Step 303: Control the surgical light to move to a target focus position according to the movement trajectory of the world coordinate system, and extract a facial image of the operator to extract a corresponding eye image from the facial image.

[0079] In this embodiment, this step mainly includes: after obtaining the world coordinate system movement trajectory of the robotic arm, through moveit path planning, trajectory points containing time information, acceleration information and speed information are obtained according to the world coordinate system movement trajectory, and spline interpolation is performed on the trajectory points for a preset number of times to obtain trajectory parameters when the robotic arm moves, and the surgical light is controlled to move according to the world coordinate system movement trajectory according to the trajectory parameters, so as to execute the reproduction of the movement trajectory of the robotic arm in the dynamic image until it moves to the target focusing position.

[0080] Furthermore, a facial image of the surgeon when the surgical light is located at the target focus position is obtained, and the facial image is processed respectively by a pre-trained facial detection model and a facial key point detection model to obtain a facial detection frame and facial key points; the face is cropped according to the facial detection frame to obtain facial image data and the facial image is cropped by the facial key points to obtain the eye image; wherein the eye image includes a left eye image and a right eye image.

[0081] Specifically, in certain implementations of the present embodiment, the facial contour in the facial image is extracted by a preset edge detection algorithm, and the facial image is cropped according to the facial contour to obtain a facial contour map corresponding to the target surgical performer, so as to avoid the influence of the background image on the key point detection.

[0082] Next, the key points of the facial contour map are marked using an existing facial key point detection algorithm such as the PFLD algorithm to obtain the facial key point information corresponding to the target surgical implementer, and the key points corresponding to the left and right pupils are used as the cropping center to connect the key points of the outer contour of each eye image to generate an eye contour line corresponding to each eye image, and the facial contour map is cropped according to the eye contour line to generate an eye image corresponding to the target surgical implementer, and the eye image includes the left and right eye images of the surgical implementer.

[0083] Step 304: Input the facial image and the eye image into a pre-trained deep learning model to extract posture features corresponding to the facial image and the eye image through the deep learning model, and then output the line of sight angle of the surgeon.

[0084] In this embodiment, the steps mainly include: performing convolution processing on the facial image and the eye image respectively through the convolution network preset in the deep learning model, extracting the first posture feature corresponding to the facial image and the second posture feature corresponding to the eye image; performing weighted summation on the first posture feature and the second posture feature through the fully connected layer preset in the deep learning model, and obtaining the sight line feature corresponding to the surgical performer; inputting the sight line feature into the classifier preset in the deep learning model, and outputting the sight line angle corresponding to the surgical performer through the classifier.

[0085] Specifically, in certain implementations of the present embodiment, the deep learning model performs convolution processing on the original facial image and eye image respectively, extracts the center point data corresponding to the facial image, and performs image segmentation on the facial image according to the center point data, dividing the image into four side images of upper, lower, left and right, and performs convolution processing on the four side images again to extract the number of black pixels corresponding to the four side images respectively, and determines the head posture characteristics of the surgeon according to the number of melanin pixels in each side image, that is, if the number of melanin pixels in the upper image is greater than that in the lower image, then the first head posture feature of the surgeon is biased upward, and if at the same time the number of melanin pixels in the right image is greater than that in the left image, then the head posture feature of the surgeon extracted at this time is biased to the upper right.

[0086] While extracting the posture features of the head, the deep learning model performs the same convolution processing on the eye image, extracts the center point data corresponding to the eye image, and performs image segmentation on the eye image according to the center point data, dividing the image into four side images of upper, lower, left and right, and performs convolution processing on the four side images again to extract the number of black pixels corresponding to the four side images respectively, and determines the head posture features of the surgeon according to the number of melanin pixels in each side image, that is, if the number of melanin pixels in the upper image is greater than that in the lower image, the eye posture feature of the surgeon is biased upward, and if the number of melanin pixels in the right image is greater than that in the left image at the same time, the eye posture feature of the surgeon is extracted as upper right.

[0087] After acquiring the eye posture features and the head posture features, weights are adaptively assigned to the eye features and the head features according to the fully connected layer preset in the deep learning model, and the weights assigned to each posture feature are cumulatively added to obtain the sight line features corresponding to the target surgical performer.

[0088] The sight line feature is input into a pre-trained classifier such as a soft classification function, etc., and the classifier finds a sight line direction that matches the sight line feature according to the sight line feature.

[0089] Step 305: Calculate the target irradiation angle of the surgical light according to the sight angle, and obtain a light spot image of the surgical area corresponding to the target irradiation angle.

[0090] In this embodiment, the step mainly includes: obtaining the head coordinate position of the surgeon to calculate the focus coordinates of the surgeon's line of sight focus point based on the head coordinate position and the line of sight angle; substituting the focus coordinates and the current target focus position of the surgical light into the preset focus angle calculation formula to obtain the target irradiation angle corresponding to the surgical light.

[0091] Specifically, in certain implementations of the present embodiment, the coordinate position information of the head of the operator is first obtained. Preferably, the head position coordinates can be obtained through images or sensors, and then the coordinate information corresponding to the line of sight focus of the operator is estimated based on the head position information and the line of sight angle. Among them, the coordinate information can be obtained through a preset coordinate calculation formula. Specifically, a spatial coordinate system is established with a certain point on the ground as the coordinate origin, and the vertical distance AA' between the operator and the focal point is determined by determining the coordinate position of the head A of the operator, and the angle θ between the line AB connecting the head A of the operator and the focal point B and AA' of the operator, and the coordinates of the focal point can be calculated. Among them, the AA' direction is the line of sight angle of the operator, and the line of sight angle is correspondingly decomposed into a pitch angle and a yaw angle, and the pitch angle is the angle θ between AB and AA'. Among them, the coordinates of point A are known to be A(xA, yA, zA). Since A' is the projection of point A on the ground, the coordinates of A' are A'(xA, yA, 0). Point B is on the ground, so the coordinates of point B are B(xB, yB, 0). The distance of AA' is zA. Combining the angle θ between AB and AA', we get the formula:

[0092]

[0093] When the surgeon's sight angle, i.e., the position coordinates of the surgeon's head, A(xA, yA, zA), A′(xA, yA, 0), and AA′ are known, the coordinates of point B, i.e., coordinates B(xB, yB, 0), can be obtained to determine the sight focus coordinates.

[0094] Furthermore, after obtaining the line of sight focus coordinates, according to the above calculation formula, it is only necessary to obtain the position coordinates corresponding to the current target focusing position of the surgical light to obtain the target illumination angle corresponding to the surgical light, and after the illumination angle of the surgical light is adjusted to the target illumination angle, the light spot image corresponding to the surgical implementation area at the target illumination angle is obtained.

[0095] Step 306: Performing a size analysis on the light spot image to adjust the irradiation focal length of the surgical light according to the result of the size analysis, and controlling the surgical light to perform focused irradiation on the surgical operation area according to the adjusted irradiation focal length.

[0096] In this embodiment, the step mainly includes: calculating the standard moment corresponding to the spot image to calculate the image center of the spot image according to the standard moment; obtaining the fitted spot center point according to the image center and a preset dynamic curve fitting algorithm for the air, and then calculating the size of the spot image according to the spot center point; determining the spot compensation value according to the size and a preset size threshold, and then adjusting the irradiation focal length of the surgical lamp according to the spot compensation value.

[0097] Specifically, in some implementations of this embodiment, the standard moment method is used to normalize the spot image; , then the standard moment Defined as: ; where p and q are integers and satisfy the conditions: p ≥ 0, p - |q| is an even number and |q| ≤ p; represents the xth row and yth column of the spot image; the standard moment of the spot image Get the center of the spot image , where the calculation expression of the center is:

[0098] in, Indicated on the horizontal axis The beam intensity at a point, Indicated on the vertical axis The beam intensity at a point, represents the intensity of the entire image beam. Next, the center point of the fitted spot can be obtained using the dynamic curve fitting algorithm for the air, and then the size of the spot image can be calculated based on the center point of the spot. The calculation formula of the spot size is as follows:

[0099] Among them, the The cross section along the axis The beam intensity value at the point represents the spot size, Indicates the center point of the fitted light spot.

[0100] After the light spot size is obtained, the light spot size is compared with a preset size threshold. When the light spot size is greater than the size threshold, the irradiation focal length of the light is increased, and when the light spot size is less than the size threshold, the irradiation focal length of the light is decreased.

[0101] The light focusing control method and system in the surgical process disclosed in this embodiment first obtains the corresponding dynamic image in the operating room to extract the movement trajectory of the robotic arm used to track the surgical process from the dynamic image, and then controls the surgical light to move according to the movement trajectory. Among them, since the robotic arm always tracks the surgical process of the operator, when the position of the surgical light is adjusted according to the movement trajectory of the robotic arm, it can be ensured that the illumination of the surgical light fits the surgical requirements of the operator without manual adjustment, thereby ensuring both the efficiency and accuracy of the surgical light adjustment.

[0102] Furthermore, after adjusting the position of the surgical light itself according to the movement of the robotic arm, the irradiation target of the surgical light also changes due to the movement of the position. In order to ensure that the irradiation direction of the surgical light is consistent with the irradiation requirements of the surgeon, the present application obtains the surgeon's line of sight to adjust the irradiation angle of the surgical light, thereby improving the accuracy of the irradiation of the surgical light. Furthermore, after adjusting the irradiation angle, in order to avoid the irradiation of the surgical light affecting the surgical field of view of the surgeon, the corresponding spot image in the current irradiation direction can be obtained, and the irradiation distance of the surgical light can be adjusted by performing a size analysis on the spot image, thereby avoiding overexposure that affects the surgical process and ensuring the effect of the adjustment.

[0103] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for controlling light focus during surgery, characterized in that: include: Acquire a corresponding dynamic image in the operating room, and acquire a pixel trajectory of the robotic arm when it moves in the dynamic image; Performing coordinate system conversion on the pixel trajectory to obtain trajectory parameters corresponding to the movement of the robot arm and controlling the surgical light to move to a target focusing position according to the trajectory parameters; Acquire a facial image of a person performing surgery when the surgical light is located at the target focus position, and extract eye key points from the facial image to extract a corresponding eye image from the facial image; Inputting the eye image and the facial image into a pre-trained deep learning model, so as to obtain the sight angle of the operator through the deep learning model, and then obtaining the target irradiation angle of the surgical lamp according to the sight angle; Acquire a light spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform a size analysis on the light spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis; The surgical lamp is controlled to perform focused illumination on the surgical operation area according to the adjusted illumination focal length.

2. A method for controlling light focus during surgery according to claim 1, characterized in that: The obtaining of the corresponding dynamic image in the operating room comprises: Splitting the dynamic image into a plurality of continuous static image frames; Performing grayscale processing on each of the static image frames to extract the outline of the robotic arm in the static image frame; wherein the robotic arm is used to track the progress of the surgery; Filtering a corresponding robotic arm region in the static image frame according to the contour to extract pixel feature information corresponding to the robotic arm region; The static image frame is divided into a plurality of sub-mechanical arm regions by using a clustering algorithm, and the sub-mechanical arm regions whose pixel feature information ratios with the mechanical arm region are within a predetermined range are merged to obtain a plurality of continuous mechanical arm image frames.

3. A method for controlling light focus during surgery according to claim 2, characterized in that: The step of obtaining a pixel trajectory of the robot arm when it moves in the dynamic image includes: Based on a plurality of the robotic arm image frames, an image matrix difference calculation is performed to obtain a difference between two preceding and following robotic arm image frames to generate a difference image; Obtaining the contour of the difference image, and saving the contour and its centroid and coordinate values; The pixel trajectory is obtained based on the centroids of the contours of all the difference images.

4. The method for controlling light focus during surgery according to claim 3, characterized in that: The coordinate system conversion of the pixel trajectory to obtain the trajectory parameters corresponding to the movement of the robot arm and controlling the surgical light to move to the target focus position according to the trajectory parameters includes: Converting the coordinate system of the pixel trajectory into a world coordinate system to obtain the world coordinate system movement trajectory of the robotic arm; By using moveit path planning, trajectory points including time information, acceleration information and speed information are obtained according to the movement trajectory of the world coordinate system, and spline interpolation is performed on the trajectory points for a preset number of times to obtain trajectory parameters when the robot moves; The surgical light is controlled to move according to the trajectory parameters according to the movement trajectory of the world coordinate system, so as to reproduce the movement trajectory of the robot arm in the dynamic image until it moves to the target focusing position.

5. The method for controlling light focus during surgery according to claim 1, characterized in that: The step of extracting eye key points from the facial image to extract a corresponding eye image from the facial image includes: Processing the facial image by using a pre-trained facial detection model and a facial key point detection model respectively to obtain a facial detection frame and facial key points; The face is cropped according to the face detection frame to obtain face image data, and the face image is cropped according to the face key points to obtain the eye image; wherein the eye image includes a left eye image and a right eye image.

6. A method for controlling light focus during surgery according to claim 5, characterized in that: The obtaining the sight angle of the operator performing the surgery through the deep learning model includes: Performing convolution processing on the facial image and the eye image respectively through the convolution network preset in the deep learning model to extract a first posture feature corresponding to the facial image and a second posture feature corresponding to the eye image; Obtaining the sight line feature corresponding to the surgical operator by performing a weighted summation of the first posture feature and the second posture feature through a fully connected layer preset in the deep learning model; The sight line feature is input into a classifier preset in the deep learning model, and the sight line angle corresponding to the surgical personnel is output through the classifier.

7. A method for controlling light focus during surgery according to claim 6, characterized in that: The step of obtaining a target irradiation angle of the surgical lamp according to the sight angle includes: Acquiring the head coordinate position of the operator to calculate the focus coordinates of the operator's line of sight focus point according to the head coordinate position and the line of sight angle; The focusing coordinates and the current target focusing position of the surgical light are substituted into a preset focusing angle calculation formula to obtain a target irradiation angle corresponding to the surgical light.

8. The method for controlling light focus during surgery according to claim 1, characterized in that: The step of performing a size analysis on the light spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis includes: Calculating a standard moment corresponding to the spot image, so as to calculate an image center of the spot image according to the standard moment; Obtaining a fitted spot center point according to the image center and a preset air dynamic curve fitting algorithm, and then calculating the size of the spot image according to the spot center point; The irradiation focal length of the surgical lamp is adjusted according to the size and a preset size threshold.

9. A light focusing control system during surgery, characterized in that: It includes a trajectory tracking module, a coordinate conversion module, an image acquisition module, a sight estimation module, an angle adjustment module and a distance adjustment module; The trajectory tracking module is used to obtain the corresponding dynamic image in the operating room, and obtain the pixel trajectory of the robot arm when it moves in the dynamic image; The coordinate conversion module is used to perform coordinate system conversion on the pixel trajectory to obtain trajectory parameters corresponding to the movement of the robot arm and control the surgical light to move to the target focus position according to the trajectory parameters; The image acquisition module is used to obtain a facial image of a surgical practitioner when the surgical light is located at the target focus position, and to extract eye key points from the facial image to extract a corresponding eye image from the facial image; The sight line estimation module is used to input the eye image and the face image into a pre-trained deep learning model, so as to obtain the sight line angle of the operator through the deep learning model, and then obtain the target irradiation angle of the surgical lamp according to the sight line angle; The angle adjustment module is used to obtain a light spot image corresponding to the surgical operation area when the surgical lamp is irradiated according to the target irradiation angle, and to perform a size analysis on the light spot image, so as to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis; The distance adjustment module is used to control the surgical lamp to perform focused illumination on the surgical operation area according to the adjusted illumination focal length.

10. A light focusing control system in a surgical procedure according to claim 9, characterized in that: The trajectory tracking module includes an image parsing unit and a trajectory building unit; The image analysis unit is used to split the dynamic image into a number of continuous static image frames; perform grayscale processing on each of the static image frames to extract the contour of the robotic arm in the static image frame; wherein the robotic arm is used to track the progress of the surgery; screen the corresponding robotic arm region in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm region; use a clustering algorithm to divide the static image frame into a plurality of sub-robotic arm regions, and merge the sub-robotic arm regions whose pixel feature information ratios with the robotic arm region are within a predetermined range to obtain a number of continuous robotic arm image frames; The trajectory building unit is used to perform image matrix difference calculation based on several of the robotic arm image frames to obtain the difference between the two preceding and succeeding robotic arm image frames to generate a difference image; obtain the contour of the difference image, save the contour and its center of mass and coordinate values; and obtain the pixel trajectory based on the center of mass of the contours of all the difference images.

Images