A method and system for controlling light focusing during a surgical procedure

Abstract

The present invention discloses a method and system for controlling the light focusing during a surgical procedure. The method includes: obtaining a corresponding dynamic image in the operating room to obtain the pixel trajectory of the robotic arm, and then controlling the surgical lamp to move to the target focusing position according to the pixel trajectory; obtaining the facial image and eye image of the surgical operator when the surgical lamp is at the target focusing position, and inputting the eye image and the facial image into a pre-trained deep learning model to obtain the line-of-sight angle of the surgical operator, and then obtaining the target irradiation angle of the surgical lamp according to the line-of-sight angle; obtaining the spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and performing size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis, so as to improve the efficiency and accuracy of the light focusing control.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and specifically, to a method and system for controlling the light focusing during a surgical procedure. Background Art

[0002] Surgical operations have extremely high requirements for operation accuracy, and the light irradiation of the surgical lamp plays a crucial role in enabling the doctor to obtain a clear field of view. Existing surgical lamps are all ceiling-mounted or floor-standing, and their irradiation angles cannot be adaptively adjusted, often relying on a third-party operator, which not only causes delays in the movement of the light, but also there may be errors in the operator's understanding of the specific irradiation effect required by the doctor, thus affecting the surgical efficiency and effect.

[0003] Furthermore, during the adjustment process of the operation, due to the different surgical sites targeted by various surgical types, when medical staff adjust the position of the surgical lamp, they also need to adjust the irradiation direction or the irradiation distance of the lamp head, etc. And this adjustment process depends on the continuous attempts of medical staff, which not only cannot meet the requirements of the operation in a timely manner, but also may cause changes in the optical parameters during the adjustment process, making the state of the optical parameters unstable, and further affecting the surgical effect.

[0004] In recent years, with the development of technology and the standardized requirements for the surgical process, robotic arms have gradually been applied to the surgical process to perform more precise surgical operations or detect the surgical process, thereby providing more accurate data basis for subsequent surgical review. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention discloses a method and system for controlling the light focusing during a surgical procedure, which is used to improve the efficiency and accuracy of light focusing control.

[0006] In order to achieve the above object, the present invention discloses a method for controlling the light focusing during a surgical procedure, including:

[0007] Obtain the corresponding dynamic image in the operating room, and obtain the pixel trajectory when the robotic arm moves in the dynamic image;

[0008] Perform coordinate system conversion on the pixel trajectory to obtain the corresponding trajectory parameters when the robotic arm moves and control the surgical lamp to move to the target focusing position according to the trajectory parameters;

[0009] Obtain the face image of the surgical implementer when the surgical lamp is at the target focusing position, and perform eye key point extraction on the face image to extract the corresponding eye image from the face image;

[0010] Input the eye image and the face image into a pre-trained deep learning model to obtain the line-of-sight angle of the surgical operator through the deep learning model, and then obtain the target irradiation angle of the surgical lamp according to the line-of-sight angle;

[0011] Obtain the spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis;

[0012] Control the surgical lamp to perform focused irradiation on the surgical operation area according to the adjusted irradiation focal length.

[0013] A method for controlling the light focusing during a surgical procedure disclosed by the present invention first obtains a corresponding dynamic image in the operating room to extract the moving trajectory of the robotic arm for tracking the surgical procedure from the dynamic image, and then controls the surgical lamp to move according to the moving trajectory. Among them, since the robotic arm always tracks the surgical procedure of the surgical operator, when adjusting the position of the surgical lamp according to the moving trajectory of the robotic arm, it can ensure that the irradiation of the surgical lamp conforms to the surgical requirements of the surgical operator, and there is no need for manual adjustment, thus ensuring both the efficiency of adjusting the surgical lamp and the accuracy of adjusting the surgical lamp.

[0014] Further, after adjusting the position of the surgical lamp itself according to the movement of the robotic arm, since the irradiation target of the surgical lamp has also changed at this time due to the movement of the position, in order to ensure that the irradiation direction of the surgical lamp is consistent with the irradiation requirements required by the surgical operator, the present application adjusts the irradiation angle of the surgical lamp by obtaining the line-of-sight angle of the surgical operator to improve the accuracy of the surgical lamp irradiation. Further, after adjusting the irradiation angle, in order to avoid the irradiation of the surgical lamp affecting the surgical field of view of the surgical operator, the corresponding spot image in the current irradiation direction can be obtained to adjust the irradiation distance of the surgical lamp by performing size analysis on the spot image, thereby avoiding overexposure of the irradiation and affecting the surgical process and ensuring the adjustment effect.

[0015] As a preferred example, the obtaining of the corresponding dynamic image in the operating room includes:

[0016] Split the dynamic image into a plurality of consecutive static image frames;

[0017] Perform gray-scale processing on each of the static image frames to extract the contour of the robotic arm in the static image frame; wherein, the robotic arm is used to track the surgical procedure;

[0018] Filter the corresponding robotic arm area in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm area;

[0019] The static image frames are divided into multiple sub - robotic - arm regions by using a clustering algorithm, and the sub - robotic - arm regions with the ratio of pixel feature information to the robotic - arm region within a predetermined range are merged to obtain several consecutive robotic - arm image frames.

[0020] After obtaining the corresponding dynamic image in the operating room, the present invention divides each static image frame in the dynamic image into regions, so as to avoid interference from other backgrounds in the image on the recognition of the movement of the robotic arm, improve the accuracy of robotic - arm movement recognition, and at the same time filter out other background pixels in the image, reduce the amount of subsequent data processing, and thus improve the efficiency of trajectory recognition, so as to improve the efficiency and accuracy of lighting control.

[0021] As a preferred example, obtaining the pixel trajectory of the robotic arm when moving in the dynamic image includes:

[0022] Based on several of the robotic - arm image frames, calculate the difference of the image matrix to obtain the difference between two consecutive robotic - arm image frames, so as to generate a difference picture;

[0023] Obtain the contour of the difference picture, and save the contour, its centroid and coordinate values;

[0024] Based on the centroids of the contours of all the difference pictures, obtain the pixel trajectory.

[0025] The present invention compares the differences between consecutive frames, and then obtains the contour of the difference picture, so as to obtain the centroid of the movement trajectory through the contour, and then uses the movement of the centroid to describe the movement trajectory of the robotic arm, ensuring the accuracy of trajectory recognition.

[0026] As a preferred example, performing coordinate system conversion on the pixel trajectory to obtain the corresponding trajectory parameters when the robotic arm moves and controlling the surgical lamp to move to the target focus position according to the trajectory parameters includes:

[0027] Convert the coordinate system of the pixel trajectory to the world coordinate system to obtain the world - coordinate - system movement trajectory of the robotic arm;

[0028] Through moveit path planning, obtain trajectory points including time information, acceleration information and speed information based on the world - coordinate - system movement trajectory, and perform spline interpolation on the trajectory points for a preset number of times to obtain the trajectory parameters when the robotic arm moves;

[0029] Control the surgical lamp to move according to the world - coordinate - system movement trajectory according to the trajectory parameters, so as to reproduce the movement trajectory of the robotic arm in the dynamic image until it moves to the target focus position.

[0030] After generating the movement trajectory of the robotic arm in the picture through the changes of pixels in the picture, in order to adjust the surgical lamp, it is also necessary to convert the movement in the image into the movement in the real world. After moving the coordinate system, the trajectory parameters of the movement trajectory are generated by path planning to ensure the accuracy of the surgical lamp adjustment according to the trajectory parameters.

[0031] As a preferred example, the extraction of eye key points from the face image to extract the corresponding eye image from the face image includes:

[0032] Processing the face image through a pre-trained face detection model and a face key point detection model respectively to obtain a face detection frame and face key points;

[0033] Cropping the face according to the face detection frame to obtain face image data and cropping the face image through the face key points to obtain the eye image; wherein, the eye image includes a left eye image and a right eye image.

[0034] When estimating the line-of-sight angle of the surgical operator, in order to ensure the accuracy of the estimation, the eye image and the face image of the surgical operator are obtained simultaneously, so as to perform synchronous pose estimation on the face image and the eye image, and improve the accuracy of the line-of-sight estimation according to the pose correction between the face and the eyes, thereby improving the accuracy of the adjustment of the irradiation angle of the surgical lamp.

[0035] As a preferred example, the obtaining of the line-of-sight angle of the surgical operator through the deep learning model includes:

[0036] Performing convolution processing on the face image and the eye image respectively through a preset convolutional network in the deep learning model to extract a first pose feature corresponding to the face image and a second pose feature corresponding to the eye image;

[0037] Performing weighted summation on the first pose feature and the second pose feature through a fully connected layer preset in the deep learning model to obtain a line-of-sight feature corresponding to the surgical operator;

[0038] Inputting the line-of-sight feature into a classifier preset in the deep learning model, and outputting the line-of-sight angle corresponding to the surgical operator through the classifier.

[0039] The present invention uses the deep learning model to fuse the pose features of the face and the eyes for line-of-sight estimation, improves the accuracy of line-of-sight estimation. Further, a fully connected layer is set in the deep learning model to perform weighted summation on the features, reducing the amount of data processing, and thus reducing the training cost of the model.

[0040] As a preferred example, obtaining the target irradiation angle of the surgical lamp according to the line-of-sight angle includes:

[0041] Obtaining the head coordinate position of the surgical operator, and calculating the focusing coordinates of the line-of-sight focus point of the surgical operator according to the head coordinate position and the line-of-sight angle;

[0042] Substituting the focusing coordinates and the current target focusing position of the surgical lamp into a preset focusing angle calculation formula to obtain the corresponding target irradiation angle of the surgical lamp.

[0043] The present invention utilizes the estimated line-of-sight angle and combines with the head position of the surgical operator to obtain the line-of-sight focus corresponding to the surgical operator, and then determines the irradiation angle of the surgical lamp according to the line-of-sight focus and the current position information of the surgical lamp, ensuring that the irradiation angle of the surgical lamp meets the focusing requirements of the surgical operator and improving the accuracy of the adjustment of the surgical lamp.

[0044] As a preferred example, performing size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis includes:

[0045] Calculating the standard moment corresponding to the spot image, and calculating the image center of the spot image according to the standard moment;

[0046] Obtaining the fitted spot center point according to the image center and a preset air dynamic curve fitting algorithm, and then calculating the size of the spot image according to the spot center point;

[0047] Determining a spot compensation value according to the size and a preset size threshold, and then adjusting the irradiation focal length of the surgical lamp according to the spot compensation value.

[0048] After adjusting the angle and position, in order to avoid overexposure of the irradiation and thus affect the surgical vision of the surgical operator, the present invention analyzes the size of the corresponding spot image under the current irradiation, and adjusts the irradiation distance of the surgical lamp according to the size, which can ensure that the surgical lamp provides a good surgical vision for the surgical operator at the irradiation distance and improve the focusing effect.

[0049] On the other hand, the present invention discloses a lighting focusing control system during a surgical procedure, including a trajectory tracking module, a coordinate conversion module, an image acquisition module, a line-of-sight estimation module, an angle adjustment module, and a distance adjustment module;

[0050] The trajectory tracking module is used to obtain the corresponding dynamic image in the operating room and obtain the pixel trajectory when the robotic arm moves in the dynamic image;

[0051] The coordinate conversion module is used to perform coordinate conversion on the pixel trajectory to obtain the corresponding trajectory parameters when the robotic arm moves, and control the surgical lamp to move to the target focusing position according to the trajectory parameters;

[0052] The image acquisition module is used to acquire the face image of the surgical operator when the surgical lamp is at the target focusing position, and extract the key points of the eyes from the face image to extract the corresponding eye image from the face image;

[0053] The line-of-sight estimation module is used to input the eye image and the face image into a pre-trained deep learning model to obtain the line-of-sight angle of the surgical operator through the deep learning model, and further obtain the target irradiation angle of the surgical lamp according to the line-of-sight angle;

[0054] The angle adjustment module is used to acquire the spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis;

[0055] The distance adjustment module is used to control the surgical lamp to perform condensing irradiation on the surgical operation area according to the adjusted irradiation focal length.

[0056] A lighting focusing control system during a surgical procedure disclosed by the present invention first acquires the corresponding dynamic image in the operating room to extract the movement trajectory of the robotic arm for tracking the surgical procedure from the dynamic image, and then controls the movement of the surgical lamp according to the movement trajectory. Among them, since the robotic arm always tracks the surgical procedure of the surgical operator, when adjusting the position of the surgical lamp according to the movement trajectory of the robotic arm, it can ensure that the irradiation of the surgical lamp conforms to the surgical requirements of the surgical operator, without the need for manual adjustment, thus ensuring both the efficiency of the surgical lamp adjustment and the accuracy of the surgical lamp adjustment.

[0057] Further, after adjusting the position of the surgical lamp itself according to the movement of the robotic arm, since the irradiation target of the surgical lamp also changes at this time due to the movement of the position, in order to ensure that the irradiation direction of the surgical lamp is consistent with the irradiation requirements of the surgical operator, the present application adjusts the irradiation angle of the surgical lamp by acquiring the line-of-sight angle of the surgical operator to improve the accuracy of the surgical lamp irradiation. Further, after adjusting the irradiation angle, in order to avoid the irradiation of the surgical lamp affecting the surgical field of the surgical operator, the spot image corresponding to the current irradiation direction can be acquired to adjust the irradiation distance of the surgical lamp by performing size analysis on the spot image, thereby avoiding overexposure of the irradiation and affecting the surgical process and ensuring the adjustment effect.

[0058] As a preferred example, the trajectory tracking module includes an image analysis unit and a trajectory construction unit;

[0059] The image analysis unit is used to split the dynamic image into a plurality of consecutive static image frames; perform grayscale processing on each of the static image frames to extract the contour of the robotic arm in the static image frame; wherein, the robotic arm is used to track the surgical process; screen the corresponding robotic arm area in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm area; use a clustering algorithm to divide the static image frames into multiple sub-robotic arm areas, and merge the sub-robotic arm areas whose ratio of pixel feature information to the robotic arm area is within a predetermined range to obtain a plurality of consecutive robotic arm image frames;

[0060] The trajectory construction unit is used to perform image matrix difference calculation based on a plurality of the robotic arm image frames to obtain the difference between two consecutive robotic arm image frames to generate a difference picture; obtain the contour of the difference picture, save the contour and its centroid and coordinate values; based on the centroids of the contours of all the difference pictures, obtain the pixel trajectory.

[0061] After obtaining the corresponding dynamic image in the operating room, the present invention divides each frame of static image in the dynamic image to avoid interference from other backgrounds in the image to the recognition of the movement of the robotic arm, improve the accuracy of the recognition of the movement of the robotic arm, and at the same time filter out other background pixels in the image to reduce the amount of subsequent data processing, thereby improving the efficiency of trajectory recognition and improving the efficiency and accuracy of light control.

[0062] Furthermore, by comparing the differences between consecutive frames, the contour of the difference picture is obtained, so that the centroid of the movement trajectory can be obtained through the contour, and then the movement trajectory of the robotic arm is described by using the movement of the centroid to ensure the accuracy of trajectory recognition. Description of the Drawings

[0063] Figure 1 : is a schematic flow chart of a method for controlling light focusing during a surgical process disclosed in an embodiment of the present invention;

[0064] Figure 2 : is a schematic structural diagram of a system for controlling light focusing during a surgical process disclosed in an embodiment of the present invention;

[0065] Figure 3 : is a schematic flow chart of a method for controlling light focusing during a surgical process disclosed in another embodiment of the present invention. Detailed Embodiments

[0066] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0067] Embodiment 1

[0068] This embodiment discloses a method for controlling the light focus during a surgical procedure. Specifically, the specific implementation process of the control method can refer to Figure 1 , and mainly includes steps 101 to 106. The steps are mainly as follows:

[0069] Step 101: Obtain the corresponding dynamic image in the operating room, and obtain the pixel trajectory when the robotic arm moves in the dynamic image.

[0070] In this embodiment, this step mainly includes: splitting the dynamic image into several consecutive static image frames; performing grayscale processing on each static image frame to extract the contour of the robotic arm in the static image frame, where the robotic arm is used to track the surgical process; screening the corresponding robotic arm area in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm area; using a clustering algorithm to divide the static image frames into multiple sub-robotic arm areas, and merging the sub-robotic arm areas whose ratio of pixel feature information to the robotic arm area is within a predetermined range to obtain several consecutive robotic arm image frames.

[0071] Further, based on several robotic arm image frames, calculate the difference of the image matrix to obtain the difference between the previous and the next robotic arm image frames, so as to generate a difference picture; obtain the contour of the difference picture, save the contour and its centroid and coordinate values; based on the centroids of the contours of all the difference pictures, obtain the pixel trajectory.

[0072] In this embodiment, after obtaining the corresponding dynamic image in the operating room, this step divides each frame of the static image in the dynamic image to avoid interference from other backgrounds in the image on the recognition of the movement of the robotic arm, improve the accuracy of the recognition of the movement of the robotic arm, and at the same time filter out other background pixels in the image to reduce the subsequent data processing volume, thereby improving the efficiency of trajectory recognition and improving the efficiency and accuracy of light control. Further, by comparing the differences between consecutive frames, the contour of the difference picture is obtained, so that the centroid of the moving trajectory can be obtained through the contour, and then the movement of the robotic arm is described by the movement of the centroid, ensuring the accuracy of trajectory recognition.

[0073] Step 102: Perform coordinate system transformation on the pixel trajectory to obtain the corresponding trajectory parameters when the robotic arm moves, and control the surgical lamp to move to the target focusing position according to the trajectory parameters.

[0074] In this embodiment, this step mainly includes: converting the coordinate system of the pixel trajectory into the world coordinate system to obtain the world coordinate system movement trajectory of the robotic arm; through moveit path planning, obtaining trajectory points including time information, acceleration information, and velocity information based on the world coordinate system movement trajectory, and performing spline interpolation on the trajectory points a preset number of times to obtain the trajectory parameters when the robotic arm moves; controlling the surgical lamp to move according to the world coordinate system movement trajectory according to the trajectory parameters to reproduce the movement trajectory of the robotic arm in the dynamic image until it moves to the target focusing position.

[0075] In this embodiment, after generating the movement trajectory of the robotic arm in the picture through the change of pixels in the picture, in order to realize the adjustment of the surgical lamp, it is also necessary to convert the movement in the image into the movement in the actual world. After performing coordinate system movement, the trajectory parameters of the movement trajectory are generated by path planning to ensure the accuracy of the surgical lamp adjustment according to the trajectory parameters.

[0076] Step 103: Obtain the facial image of the surgical operator when the surgical lamp is at the target focusing position, and perform eye key point extraction on the facial image to extract the corresponding eye image from the facial image.

[0077] In this embodiment, this step mainly includes: processing the facial image through a pre-trained face detection model and a face key point detection model respectively to obtain a face detection frame and face key points; cropping the face according to the face detection frame to obtain face image data and cropping the facial image according to the face key points to obtain the eye image; wherein, the eye image includes a left eye image and a right eye image.

[0078] In this embodiment, when estimating the line-of-sight angle of the surgical operator, in order to ensure the accuracy of the estimation, the eye image and the facial image of the surgical operator are obtained simultaneously to perform synchronous pose estimation on the facial image and the eye image, and improve the accuracy of the line-of-sight estimation according to the pose correction between the face and the eyes, thereby improving the accuracy of the adjustment of the irradiation angle of the surgical lamp.

[0079] Step 104: Input the eye image and the facial image into a pre-trained deep learning model to obtain the line-of-sight angle of the surgical operator through the deep learning model, and then obtain the target irradiation angle of the surgical lamp according to the line-of-sight angle.

[0080] In this embodiment, this step mainly includes: respectively performing convolution processing on the face image and the eye image through a preset convolutional network in the deep learning model to extract a first pose feature corresponding to the face image and a second pose feature corresponding to the eye image; performing weighted summation on the first pose feature and the second pose feature through a fully connected layer preset in the deep learning model to obtain a gaze feature corresponding to the surgical operator; inputting the gaze feature into a classifier preset in the deep learning model, and outputting a gaze angle corresponding to the surgical operator through the classifier.

[0081] Further, obtain the head coordinate position of the surgical operator to calculate the focus coordinate of the gaze focus of the surgical operator according to the head coordinate position and the gaze angle; substitute the focus coordinate and the current target focus position of the surgical lamp into a preset focus angle calculation formula to obtain the target irradiation angle corresponding to the surgical lamp.

[0082] In this embodiment, this step uses the deep learning model to fuse the pose features of the face and eyes for gaze estimation, improving the accuracy of gaze estimation. Further, a fully connected layer is set in the deep learning model to perform weighted summation on the features, reducing the amount of data processing, and thus reducing the training cost of the model. Further, using the estimated gaze angle and combining with the head position of the surgical operator to obtain the gaze focus corresponding to the surgical operator, and then determining the irradiation angle of the surgical lamp according to the gaze focus and the current position information of the surgical lamp, ensuring that the irradiation angle of the surgical lamp meets the focusing requirements of the surgical operator and improving the accuracy of surgical lamp adjustment.

[0083] Step 105: Obtain a spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis.

[0084] In this embodiment, this step mainly includes: calculating the standard moment corresponding to the spot image to calculate the image center of the spot image according to the standard moment; obtaining the fitted spot center point according to the image center and a preset empty dynamic curve fitting algorithm, and then calculating the size of the spot image according to the spot center point; determining a spot compensation value according to the size and a preset size threshold, and then adjusting the irradiation focal length of the surgical lamp according to the spot compensation value.

[0085] In this embodiment, after adjusting the angle and position, in order to avoid overexposure of the irradiation and thus affect the surgical vision of the surgical operator, the size of the corresponding spot image under the current irradiation is analyzed, and the irradiation distance of the surgical lamp is adjusted according to the size, so as to ensure that the surgical lamp provides a good surgical vision for the surgical operator at the irradiation distance and improve the focusing effect.

[0086] Step 106: Control the surgical lamp to perform focused irradiation on the surgical operation area according to the adjusted irradiation focal length.

[0087] On the other hand, this embodiment also discloses a lighting focusing control system during a surgical procedure. For the specific structural composition of the control system, please refer to Figure 2 , including a trajectory tracking module 201, a coordinate conversion module 202, an image acquisition module 203, a line-of-sight estimation module 204, an angle adjustment module 205, and a distance adjustment module 206.

[0088] The trajectory tracking module 201 is used to obtain the corresponding dynamic image in the operating room and obtain the pixel trajectory when the robotic arm moves in the dynamic image.

[0089] The coordinate conversion module 202 is used to perform coordinate system conversion on the pixel trajectory to obtain the corresponding trajectory parameters when the robotic arm moves and control the surgical lamp to move to the target focusing position according to the trajectory parameters.

[0090] The image acquisition module 203 is used to obtain the face image of the surgical operator when the surgical lamp is at the target focusing position, and perform eye key point extraction on the face image to extract the corresponding eye image from the face image.

[0091] The line-of-sight estimation module 204 is used to input the eye image and the face image into a pre-trained deep learning model to obtain the line-of-sight angle of the surgical operator through the deep learning model, and then obtain the target irradiation angle of the surgical lamp according to the line-of-sight angle.

[0092] The angle adjustment module 205 is used to obtain the spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis.

[0093] The distance adjustment module 206 is used to control the surgical lamp to perform focused irradiation on the surgical operation area according to the adjusted irradiation focal length.

[0094] In this embodiment, the trajectory tracking module 201 includes an image analysis unit and a trajectory construction unit.

[0095] The image analysis unit is used to split the dynamic image into a plurality of consecutive static image frames; perform grayscale processing on each of the static image frames to extract the contour of the robotic arm in the static image frame, wherein the robotic arm is used to track the surgical process; screen the corresponding robotic arm area in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm area; use a clustering algorithm to divide the static image frames into multiple sub-robotic arm areas, and merge the sub-robotic arm areas whose ratio of pixel feature information to the robotic arm area is within a predetermined range to obtain a plurality of consecutive robotic arm image frames.

[0096] The trajectory construction unit is used to perform image matrix difference calculation based on a plurality of the robotic arm image frames to obtain the difference between two consecutive robotic arm image frames, so as to generate a difference picture; obtain the contour of the difference picture, and save the contour, its centroid and coordinate values; based on the centroids of the contours of all the difference pictures, obtain the pixel trajectory.

[0097] Embodiment 2

[0098] In the traditional surgical process, the position, angle, etc. of the light are manually adjusted to provide a surgical field of view for the surgical personnel. However, the technology relying on manual light adjustment cannot timely and effectively adjust the angle and distance of light focusing in case of special situations, thus affecting the surgical process.

[0099] In response to this, in some embodiments of this embodiment, the medical robotic arm in the surgical process is used as a reference point for light adjustment, and by monitoring the movement vector of the medical robotic arm, the movement of the light is controlled to achieve effective real-time light adjustment.

[0100] Specifically, referring to Figure 3 , which is a schematic flowchart of a method for controlling light focusing during a surgical process provided in this embodiment, mainly includes steps 301 to 306, and the steps are mainly as follows:

[0101] Step 301: Obtain the monitoring video in the operating room, and perform instance segmentation on the monitoring video to obtain a dynamic image containing only the medical robotic arm.

[0102] In this embodiment, this step mainly includes: splitting the dynamic image into a number of consecutive static image frames; performing grayscale processing on each of the static image frames to extract the contour of the robotic arm in the static image frame, where the robotic arm is used to track the surgical process; screening the corresponding robotic arm area in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm area; using a clustering algorithm to divide the static image frames into multiple sub-robotic arm areas, and merging the sub-robotic arm areas whose ratio of pixel feature information to the robotic arm area is within a predetermined range to obtain a number of consecutive robotic arm image frames.

[0103] Specifically, in some embodiments of this embodiment, first split the real-time monitored video into consecutive image frames along the time axis direction, and then perform grayscale processing on each of the image frames to obtain the contour of the robotic arm. Among them, the extraction of the contour can be implemented by using the classic canny algorithm or the open-source OpenCV model, etc.

[0104] Further, after extracting the contour, intercept the area where the robotic arm is located from the image frame along the contour, and then statistically analyze the pixel feature information of the area where the robotic arm is located, so as to further screen the image frame according to the pixel feature information and improve the accuracy of robotic arm extraction.

[0105] Specifically, in the process of calculating the pixel feature information, first count the pixel points in the area where the robotic arm is located, and then calculate the central position of the area to obtain the chromaticity, saturation and intensity of the pixels within the central position, and then obtain the color feature of the area.

[0106] In some embodiments of this embodiment, the calculation formula for counting the pixel points is as follows:

[0107]

[0108] Among them, S represents the total number of pixel points that meet the conditions, W represents the length of the area, H represents the width of the area, and represents the pixel at any point within the area, and the pixel is not a white pixel.

[0109] Count all the pixel points in the area according to the calculation formula to obtain the total number of pixels in the area, and then calculate the pixel mean value in the area according to the total number of pixels.

[0110] Further, when calculating the central position of the area, calculate the position information corresponding to the central area through a preset central area calculation formula. The expression of the central area calculation formula is:

[0111]

[0112] Among them, the represents the upper left corner coordinates corresponding to the central region, and the represents the lower right corner coordinates corresponding to the central region.

[0113] Immediately calculate the chromaticity, saturation, and intensity of the pixels corresponding to the central region. Specifically, the calculation formulas for the chromaticity, saturation, and intensity are as follows:

[0114]

[0115] Among them, the represents the average value of the chromaticity C within the central region; the represents the average value of the intensity I within the central region; represents the average value of the saturation SAT within the central region; the S represents the total number of pixels meeting the conditions.

[0116] After obtaining the chromaticity, saturation, and intensity of the pixels within the central position, and thus obtaining the color features of the region, then use the clustering algorithm to divide each of the image frames into multiple sub - graphs, thus obtaining multiple sub - regions, and then use the same color feature extraction algorithm to obtain the color features corresponding to each sub - region, and merge the sub - robotic arm regions whose ratio of pixel feature information to that of the central region is within a predetermined range to obtain several consecutive robotic arm image frames.

[0117] Step 302: Obtain the pixel movement trajectory when the robotic arm moves in the dynamic image, and perform coordinate system conversion on the pixel movement trajectory to generate the world coordinate system movement trajectory of the surgical lamp.

[0118] In this embodiment, this step mainly includes: based on several of the robotic arm image frames, perform image matrix difference calculation to obtain the difference between two consecutive robotic arm image frames, so as to generate a difference picture, obtain the contour of the difference picture, save the contour and its centroid and coordinate values, and obtain the pixel trajectory based on the centroids of the contours of all the difference pictures.

[0119] Specifically, in some embodiments of this embodiment, based on several static image frames, perform image matrix difference calculation to obtain the difference between two consecutive static image frames, and generate a new difference picture. Based on the centroids of the contours of all the difference pictures, obtain the pixel coordinate system trajectory. Immediately convert the pixel coordinate system trajectory into a world coordinate system trajectory. Specifically, first convert the pixel coordinate system into a camera coordinate system, and then according to the calibration process of the camera, convert the coordinate system of the camera into a world coordinate. Among them, the relationship between the pixel coordinate system and the camera coordinate system is as follows:

[0120]

[0121] Among them, is the coordinate of the centroid in the pixel coordinate system, is the camera coordinate system, and the is the pixel coordinate system of the center point of the picture, and the represents the width of a unit pixel in the camera internal parameters.

[0122] Step 303: Control the surgical lamp to move to the target focusing position according to the movement trajectory in the world coordinate system, and extract the face image of the surgical operator to extract the corresponding eye image from the face image.

[0123] In this embodiment, this step is mainly: after obtaining the movement trajectory of the robotic arm in the world coordinate system, through moveit path planning, trajectory points including time information, acceleration information, and speed information are obtained according to the movement trajectory in the world coordinate system, and the trajectory points are subjected to spline interpolation for a preset number of times to obtain the trajectory parameters when the robotic arm moves. According to the trajectory parameters, control the surgical lamp to move according to the movement trajectory in the world coordinate system to execute the reproduction of the movement trajectory of the robotic arm in the dynamic image until it moves to the target focusing position.

[0124] Further, obtain the face image of the surgical operator when the surgical lamp is at the target focusing position, and process the face image through a pre-trained face detection model and a face key point detection model respectively to obtain a face detection frame and face key points; crop the face according to the face detection frame to obtain face image data and crop the face image through the face key points to obtain the eye image; wherein, the eye image includes a left eye image and a right eye image.

[0125] Specifically, in some embodiments of this embodiment, the face contour in the face image is extracted through a preset edge detection algorithm, and the face image is cropped according to the face contour to obtain the face contour map corresponding to the target surgical operator, so as to avoid the influence of the background image on the key point detection.

[0126] Then, use an existing face key point detection algorithm such as the PFLD algorithm to mark the key points of the face contour map to obtain the face key point information corresponding to the target surgical operator, and use the key points corresponding to the left and right pupils as the cropping center, connect the key points of the outer contour of each eye image to generate the eye contour line corresponding to each eye image, and crop the face contour map according to the eye contour line to generate the eye image corresponding to the target surgical operator, and the eye image includes the left and right eye images of the surgical operator.

[0127] Step 304: Input the face image and the eye image into a pre-trained deep learning model to extract the pose features corresponding to the face image and the eye image through the deep learning model, and then output the line-of-sight angle of the surgical operator.

[0128] In this embodiment, this step mainly includes: performing convolution processing on the face image and the eye image respectively through a preset convolutional network in the deep learning model to extract the first pose feature corresponding to the face image and the second pose feature corresponding to the eye image; performing weighted summation on the first pose feature and the second pose feature through a fully connected layer preset in the deep learning model to obtain the line-of-sight feature corresponding to the surgical operator; inputting the line-of-sight feature into a classifier preset in the deep learning model, and outputting the line-of-sight angle corresponding to the surgical operator through the classifier.

[0129] Specifically, in some embodiments of this embodiment, the deep learning model performs convolution processing on the original face image and the eye image respectively, extracts the center point data corresponding to the face image, and performs image segmentation on the face image according to the center point data, dividing it into upper, lower, left, and right side images, and performing convolution processing on the upper, lower, left, and right side images again to extract the number of black pixel points corresponding to each of the upper, lower, left, and right side images. Determine the head pose feature of the surgical operator according to the number of melanin pixel points in each side image, that is, if the number of melanin pixel points in the upper side image is greater than that in the lower side, the first head pose feature of the surgical operator is upward. If at the same time the number of melanin pixel points in the right side image is greater than that in the left side, then the head pose feature of the surgical operator is extracted as upper right at this time.

[0130] While extracting the pose feature of the head, the deep learning model performs the same convolution processing on the eye image, extracts the center point data corresponding to the eye image, and performs image segmentation on the eye image according to the center point data, dividing it into upper, lower, left, and right side images, and performing convolution processing on the upper, lower, left, and right side images again to extract the number of black pixel points corresponding to each of the upper, lower, left, and right side images. Determine the head pose feature of the surgical operator according to the number of melanin pixel points in each side image, that is, if the number of melanin pixel points in the upper side image is greater than that in the lower side, the eye pose feature of the surgical operator is upward. If at the same time the number of melanin pixel points in the right side image is greater than that in the left side, then the eye pose feature of the surgical operator is extracted as upper right at this time.

[0131] After obtaining the eye pose feature and the head pose feature, adaptively assign weights to the eye feature and the head feature according to the fully connected layer preset in the deep learning model, and perform cumulative addition according to the weights assigned to each pose feature to obtain the gaze feature corresponding to the target surgical operator.

[0132] Input the gaze feature into a pre-trained classifier such as a soft classification function, etc., and the classifier finds the matching gaze direction according to the gaze feature.

[0133] Step 305: Calculate the target irradiation angle of the surgical lamp according to the gaze angle, and obtain the spot image of the surgical area corresponding to the target irradiation angle.

[0134] In this embodiment, this step mainly includes: obtaining the head coordinate position of the surgical operator, and calculating the focusing coordinates of the gaze focus point of the surgical operator according to the head coordinate position and the gaze angle; substituting the focusing coordinates and the current target focusing position of the surgical lamp into a preset focusing angle calculation formula to obtain the target irradiation angle corresponding to the surgical lamp.

[0135] Specifically, in some embodiments of this embodiment, first obtain the head coordinate position information of the surgical operator. Preferably, the head position coordinates can be obtained through an image or a sensor, etc., and then estimate the coordinate information corresponding to the gaze focus point of the surgical operator according to the head position information and the gaze angle. Among them, the coordinate information can be obtained through a preset coordinate calculation formula. Specifically, establish a space coordinate system with a certain point on the ground as the coordinate origin, and determine the vertical distance AA' between the surgical operator's head A and the focus point by determining the coordinate position of the surgical operator's head A, as well as the angle θ between the line AB connecting the surgical operator's head A and the focus point B and AA'. Then the focus point coordinates can be calculated. Among them, the direction of AA' is the gaze angle of the surgical operator, and this gaze angle is correspondingly decomposed into a pitch angle and a yaw angle, and the pitch angle is the angle θ between AB and AA'. Among them, the coordinates of point A are known as A(xA, yA, zA). Since A' is the projection of point A on the ground, the coordinates of A' are A'(xA, yA, 0). Point B is on the ground, so the coordinates of point B are B(xB, yB, 0). The distance of AA' is zA. Combining the angle θ between AB and AA' gives the formula:

[0136]

[0137]

[0138] Given the viewing angle of the surgical operator, i.e., the position coordinates of the head of the surgical operator and A(xA, yA, zA) and A′(xA, yA, 0) as well as AA′, the coordinates of point B, i.e., coordinate B(xB, yB, 0), can be obtained to determine the viewing focus coordinates.

[0139] Further, after obtaining the viewing focus coordinates, according to the above calculation formula, only by obtaining the position coordinates corresponding to the target focusing position where the surgical lamp is currently located, the target irradiation angle corresponding to the surgical lamp can be obtained. After adjusting the irradiation angle of the surgical lamp to the target irradiation angle, the spot image corresponding to the surgical operation area at the target irradiation angle is acquired.

[0140] Step 306: Perform size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis, and control the surgical lamp to perform focused irradiation on the surgical operation area according to the adjusted irradiation focal length.

[0141] In this embodiment, this step mainly includes: calculating the standard moment corresponding to the spot image to calculate the image center of the spot image according to the standard moment; obtaining the fitted spot center point according to the image center and the preset empty dynamic curve fitting algorithm, and then calculating the size of the spot image according to the spot center point; determining the spot compensation value according to the size and the preset size threshold, and then adjusting the irradiation focal length of the surgical lamp according to the spot compensation value.

[0142] Specifically, in some embodiments of this embodiment, the method of standard moment is used to normalize the spot image; for the spot image , then the standard moment is defined as: ; where p and q are integers and satisfy the conditions: p≥0, p - |q| is even and |q|≤p; represents the x-th row and y-th column of the spot image; from the standard moment of the spot image, the center of the spot image is obtained, where the calculation expression of the center is:

[0143]

[0144] where, represents the beam intensity at the point on the horizontal axis , represents the beam intensity at the point on the vertical axis , It represents the intensity of the entire image beam. Subsequently, the center point of the spot after fitting can be obtained by using the dynamic curve fitting algorithm for the sky, and then the size of the spot image can be calculated based on the center point of the spot. The calculation formula for the spot size is as follows:

[0145]

[0146] Among them, the is the beam intensity value at the point on the cross-section along the axis, the represents the spot size, and the represents the center point of the spot after fitting.

[0147] After obtaining the spot size, compare the spot size with a preset size threshold. When the spot size is greater than the size threshold, increase the irradiation focal length of the light, and when the spot size is less than the size threshold, decrease the irradiation focal length of the light.

[0148] The method and system for controlling the light focus during the surgical process disclosed in this embodiment first obtain the corresponding dynamic image in the operating room to extract the movement trajectory of the robotic arm for tracking the surgical process from the dynamic image, and then control the movement of the surgical lamp according to the movement trajectory. Among them, since the robotic arm always tracks the surgical process of the surgical personnel, when adjusting the position of the surgical lamp according to the movement trajectory of the robotic arm, it can ensure that the irradiation of the surgical lamp fits the surgical requirements of the surgical personnel without manual adjustment, thus ensuring both the efficiency and accuracy of the adjustment of the surgical lamp.

[0149]

[0150] Further, after adjusting the position of the surgical lamp itself according to the movement of the robotic arm, since the irradiation target of the surgical lamp has changed at this time due to the movement of the position, in order to ensure that the irradiation direction of the surgical lamp is consistent with the irradiation requirements of the surgical personnel, this application adjusts the irradiation angle of the surgical lamp by obtaining the line-of-sight angle of the surgical personnel to improve the accuracy of the surgical lamp irradiation. Further, after adjusting the irradiation angle, in order to avoid the irradiation of the surgical lamp affecting the surgical field of the surgical personnel, the corresponding spot image in the current irradiation direction can be obtained to adjust the irradiation distance of the surgical lamp by analyzing the size of the spot image, thereby avoiding overexposure of the irradiation and affecting the surgical process and ensuring the adjustment effect.​The specific embodiments described above further elaborate on the objective, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only for the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. In particular, it is pointed out that for those skilled in the art, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for controlling the light focusing during a surgical process, characterized in that, Including: Obtain the corresponding dynamic image in the operating room, and obtain the pixel trajectory when the robotic arm moves in the dynamic image; Perform coordinate transformation on the pixel trajectory to obtain the corresponding trajectory parameters when the robotic arm moves, and control the surgical lamp to move to the target focusing position according to the trajectory parameters; Obtain the face image of the surgical personnel when the surgical lamp is at the target focusing position, and extract the eye key points from the face image to extract the corresponding eye image from the face image; Input the eye image and the face image into a pre-trained deep learning model to obtain the line-of-sight angle of the surgical personnel through the deep learning model, and then obtain the target irradiation angle of the surgical lamp according to the line-of-sight angle; Obtain the corresponding spot image of the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis; Control the surgical lamp to perform focused irradiation on the surgical operation area according to the adjusted irradiation focal length; Obtaining the line-of-sight angle of the surgical personnel through the deep learning model includes: Perform convolution processing on the face image and the eye image respectively through the preset convolutional network in the deep learning model, and extract the first pose feature corresponding to the face image and the second pose feature corresponding to the eye image; Perform weighted summation on the first pose feature and the second pose feature through the fully connected layer preset in the deep learning model to obtain the line-of-sight feature corresponding to the surgical personnel; Input the line-of-sight feature into the classifier preset in the deep learning model, and output the line-of-sight angle corresponding to the surgical personnel through the classifier; The deep learning model performs convolution processing on the face image and the eye image respectively, extracts the center point data corresponding to the face image, and performs image segmentation on the face image according to the center point data, segmenting it into upper, lower, left, and right side images, and performing convolution processing on the upper, lower, left, and right side images again, extracting the number of black pixel points corresponding to the upper, lower, left, and right side images respectively, and determining the head pose feature of the surgical personnel according to the number of melanin pixel points on each side image; The size analysis of the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis includes: Calculate the standard moment corresponding to the spot image to calculate the image center of the spot image according to the standard moment; Obtain the fitted spot center point according to the image center and the preset empty dynamic curve fitting algorithm, and then calculate the size of the spot image according to the spot center point; Adjust the irradiation focal length of the surgical lamp according to the size and the preset size threshold; The obtaining of the corresponding dynamic image in the operating room includes: Split the dynamic image into several consecutive static image frames; Perform grayscale processing on each static image frame to extract the contour of the robotic arm in the static image frame; wherein, the robotic arm is used to track the surgical process; Screen the corresponding robotic arm area in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm area; Use a clustering algorithm to divide the static image frame into multiple sub-robotic arm areas, and merge the sub-robotic arm areas whose pixel feature information ratio to the robotic arm area is within a predetermined range to obtain several consecutive robotic arm image frames; The obtaining of the pixel trajectory when the robotic arm moves in the dynamic image includes: Based on several of the robotic arm image frames, perform image matrix difference calculation to obtain the difference between two consecutive robotic arm image frames, so as to generate a difference picture; Obtain the contour of the difference picture, and save the contour, its centroid and coordinate values; Based on the centroids of the contours of all the difference pictures, obtain the pixel trajectory; The obtaining of the target irradiation angle of the surgical lamp according to the line-of-sight angle includes: Obtain the head coordinate position of the surgical operator, so as to calculate the focusing coordinates of the line-of-sight focus point of the surgical operator according to the head coordinate position and the line-of-sight angle; Substitute the focusing coordinates and the current target focusing position of the surgical lamp into a preset focusing angle calculation formula to obtain the target irradiation angle corresponding to the surgical lamp.

2. The method for controlling the light focusing during a surgical procedure according to claim 1, wherein, The performing of coordinate system conversion on the pixel trajectory to obtain the corresponding trajectory parameters when the robotic arm moves and controlling the surgical lamp to move to the target focusing position according to the trajectory parameters includes: Convert the coordinate system of the pixel trajectory into a world coordinate system to obtain the world coordinate system movement trajectory of the robotic arm; Through moveit path planning, obtain trajectory points including time information, acceleration information and speed information according to the world coordinate system movement trajectory, and perform spline interpolation on the trajectory points for a preset number of times to obtain the trajectory parameters when the robotic arm moves; Control the surgical lamp to move according to the world coordinate system movement trajectory according to the trajectory parameters to perform the reproduction of the movement trajectory of the robotic arm in the dynamic image until it moves to the target focusing position.

3. The method for controlling the light focusing during a surgical procedure according to claim 1, wherein, The performing of eye key point extraction on the face image to extract the corresponding eye image from the face image includes: Process the face image through a pre-trained face detection model and a face key point detection model respectively to obtain a face detection frame and face key points; Crop the face according to the face detection frame to obtain face image data and crop the face image according to the face key points to obtain the eye image; wherein, the eye image includes a left eye image and a right eye image.

4. A lighting focusing control system during a surgical procedure, characterized in that, The lighting focusing control system is used to implement the lighting focusing control method as described in claim 1, and includes a trajectory tracking module, a coordinate conversion module, an image acquisition module, a line-of-sight estimation module, an angle adjustment module and a distance adjustment module; The trajectory tracking module is used to obtain the corresponding dynamic image in the operating room and obtain the pixel trajectory when the robotic arm moves in the dynamic image; The coordinate conversion module is used to perform coordinate conversion on the pixel trajectory to obtain the corresponding trajectory parameters when the robotic arm moves and control the surgical lamp to move to the target focusing position according to the trajectory parameters; The image acquisition module is used to acquire the facial image of the surgical operator when the surgical lamp is at the target focusing position, and perform eye key point extraction on the facial image to extract the corresponding eye image from the facial image; The gaze estimation module is used to input the eye image and the facial image into a pre-trained deep learning model to obtain the gaze angle of the surgical operator through the deep learning model, and further obtain the target irradiation angle of the surgical lamp according to the gaze angle; The angle adjustment module is used to acquire the spot image corresponding to the surgical operation area when the surgical lamp irradiates according to the target irradiation angle, and perform size analysis on the spot image to adjust the irradiation focal length of the surgical lamp according to the result of the size analysis; The distance adjustment module is used to control the surgical lamp to perform focused irradiation on the surgical operation area according to the adjusted irradiation focal length; The trajectory tracking module includes an image parsing unit and a trajectory building unit; The image parsing unit is used to split the dynamic image into a plurality of consecutive static image frames; perform gray processing on each static image frame to extract the contour of the robotic arm in the static image frame; wherein, the robotic arm is used to track the surgical process; screen the corresponding robotic arm area in the static image frame according to the contour to extract the pixel feature information corresponding to the robotic arm area; use a clustering algorithm to divide the static image frames into multiple sub-robotic arm areas, and merge the sub-robotic arm areas whose pixel feature information ratio to the robotic arm area is within a predetermined range to obtain a plurality of consecutive robotic arm image frames; The trajectory building unit is used to perform image matrix difference calculation based on a plurality of the robotic arm image frames to obtain the difference between the previous and the next robotic arm image frames to generate a difference picture; obtain the contour of the difference picture, save the contour and its centroid and coordinate values; obtain the pixel trajectory based on the centroids of all the contours of the difference pictures; Obtaining the target irradiation angle of the surgical lamp according to the gaze angle includes: Obtaining the head coordinate position of the surgical operator to calculate the focusing coordinates of the gaze focus point of the surgical operator according to the head coordinate position and the gaze angle; Substitute the focusing coordinates and the current target focusing position of the surgical lamp into a preset focusing angle calculation formula to obtain the target irradiation angle corresponding to the surgical lamp.

Images