Surgical instrument control method, system, head-mounted control module

Abstract

The application relates to a surgical instrument control method, system and head-mounted control module. The surgical instrument control method comprises the following steps: acquiring eye image data and head movement data; acquiring a current control mode of the head-mounted control module; acquiring target control data according to the current control mode, and controlling the movement of the surgical instrument according to the target control data, wherein the target control data comprises at least one of eye movement data and head movement data, and the eye movement data is acquired according to the eye image data. The surgical instrument control method can simplify the control operation of the surgical instrument.

Description

Technical Field

[0001] This application relates to the field of surgical instrument technology, and in particular to a surgical instrument control method, system, and head-mounted control module. Background Technology

[0002] Currently, the control of flexible surgical instruments is mainly achieved by sending commands to the surgical carriage through the surgical instrument foot pedal and left and right master hands on the surgeon's table, thereby controlling the position and posture of the flexible surgical instruments. At the same time, the image processing host processes the images acquired by the flexible surgical instruments and performs functions such as distortion correction, white balance, image magnification / reduction, and image saving.

[0003] However, controlling the position of surgical instruments by using the doctor's foot pedal and main hand requires coordination of the hands and feet, and also requires repeatedly switching control of the instruments, making the operation quite cumbersome. Summary of the Invention

[0004] Therefore, it is necessary to provide a surgical instrument control method, system, and head-mounted control module that can simplify the control operation of surgical instruments in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a surgical instrument control method applied to a head-mounted control module; the surgical instrument control method includes:

[0006] Acquire eye image data and head motion data;

[0007] Obtain the current control mode of the head-mounted control module;

[0008] The target control data is acquired according to the current control mode, and the movement of the surgical instruments is controlled according to the target control data. The target control data includes at least one of eye movement data and head movement data, and the eye movement data is acquired based on eye image data.

[0009] In one embodiment, the eye movement data includes pupil center coordinates and three-dimensional matrix data characterizing head posture; acquiring target control data according to the current control mode and controlling the movement of surgical instruments according to the target control data includes:

[0010] When the current control mode is the first comprehensive control mode, the user's current pupil center coordinates are expanded and multiplied with the three-dimensional matrix data representing the current head posture to obtain the target one-dimensional matrix;

[0011] The target one-dimensional matrix is ​​input into the trained motion control model, and the movement of the surgical instruments is controlled according to the output data of the motion control model.

[0012] The motion control model is obtained through the following steps:

[0013] Obtain the first sample set, which includes a one-dimensional sample matrix and training labels corresponding to the one-dimensional sample matrix representing the direction of movement of the surgical instruments. The one-dimensional sample matrix is ​​obtained by multiplying the pupil center coordinates of the sample with the three-dimensional sample matrix data.

[0014] Input the one-dimensional matrix of the sample into the neural network model, and output the three-dimensional direction value corresponding to the one-dimensional matrix of the sample;

[0015] A neural network model is trained based on the label values ​​and three-dimensional orientation values ​​to obtain a motion control model.

[0016] In one embodiment, acquiring eye image data includes:

[0017] The system acquires the user's facial image and inputs it into a trained eye detection model to obtain the user's eye image data.

[0018] The eye detection model is obtained through the following steps:

[0019] Obtain a second sample set, which includes sample face images and training labels corresponding to the sample face images. The training labels are rectangular boxes that enclose the positions of the eyeballs.

[0020] Input the sample face image into the object detection model, and output the predicted image with the prediction box;

[0021] The loss function is determined based on the predicted image and training labels;

[0022] The target detection model is trained based on the training labels and loss function to obtain the eye detection model.

[0023] In one embodiment, the surgical instrument control method further includes:

[0024] Grayscale values ​​are calculated from eye image data to determine the coordinates of the eyeball center, and eye movement data is determined based on the coordinates of the eyeball center.

[0025] In one embodiment, target control data is acquired based on the current control mode, and the movement of surgical instruments is controlled based on the target control data, including:

[0026] When the current control mode is the second comprehensive control mode, the current direction of the user's eye movement is obtained based on eye movement data;

[0027] Based on head motion data, obtain the user's head rotation direction, head rotation angle, and movement duration relative to a reference posture;

[0028] Determine whether the user's current posture is a valid controlled posture based on the duration of the action;

[0029] When the current posture is an effective control posture, the rotation direction of the surgical instruments is determined based on the direction of eye movement and head rotation, and the rotation angle of the surgical instruments is determined based on the head rotation angle.

[0030] In one embodiment, target control data is acquired based on the current control mode, and the movement of surgical instruments is controlled based on the target control data, including:

[0031] When the current control mode is eye control mode, the user's pupil center position is changed based on the user's eye image data to obtain pupil data;

[0032] Obtain eye movement data based on pupil data;

[0033] The movement of surgical instruments is controlled based on eye movement data, where eye movement data is the target control data.

[0034] In one embodiment, target control data is acquired based on the current control mode, and the movement of surgical instruments is controlled based on the target control data, including:

[0035] When the current control mode is head control mode, acquire head motion data;

[0036] The movement of surgical instruments is controlled based on head motion data, where head motion data is the target control data.

[0037] In one embodiment, the motion commands include position commands and posture commands; the head control modes include position adjustment modes and posture adjustment modes; and the head motion data includes head translation data and head rotation data. The surgical instrument control method further includes:

[0038] When the current control mode is position adjustment mode, acquire head translation data for head motion data;

[0039] The surgical instruments are controlled to perform translational movements based on head translation data.

[0040] When the current control mode is posture adjustment mode, the posture of the surgical instruments is controlled based on head rotation data.

[0041] In one embodiment, the surgical instrument control method further includes:

[0042] When the current control mode is attitude adjustment mode, obtain the user's current attitude relative to the reference attitude, rotation direction, rotation angle and action holding time;

[0043] Determine whether the user's current posture is a valid controlled posture based on the duration of the action;

[0044] When the current posture is a valid control posture, output posture commands based on the rotation direction and rotation angle.

[0045] Secondly, this application provides a surgical instrument control system, comprising:

[0046] The first acquisition module is used to acquire eye image data and head motion data;

[0047] The second acquisition module is used to acquire the current control mode of the head-mounted control module;

[0048] The control module is used to acquire target control data according to the current control mode and control the movement of surgical instruments according to the target control data. The target control data includes at least one of eye movement data and head movement data, and the eye movement data is acquired based on eye image data.

[0049] Thirdly, this application provides a head-mounted control module, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the surgical instrument control method described above.

[0050] The aforementioned surgical instrument control method, system, head-mounted control module, and computer-readable storage medium, after acquiring the user's eye image data and head movement data, obtain the current control mode, acquire target control data based on the current control mode, and control the lens movement of the surgical instrument according to the target control data, thereby achieving positional adjustment of the surgical instrument. Since the acquisition of eye image data and head movement data depends on the user's eye and / or head movements, it does not require the user's hand and foot coordination, thus simplifying the control operation of the surgical instrument and making it more convenient for the user. Furthermore, target control data is only acquired based on the current control mode when the corresponding control mode is entered. When the control mode is not entered, the user's eye and head movements will not issue movement commands. By setting control modes, the probability of erroneous movement commands can be reduced. The target control data includes at least one of eye movement data and head movement data. Since eye movements are relatively rapid and head movements are relatively stable and reliable, each control mode has different control characteristics. The control mode can be selected according to the application scenario to meet the usage requirements of various application scenarios. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a surgical instrument control method in one embodiment;

[0052] Figure 2 This is a schematic diagram showing the location of the region of interest in one embodiment;

[0053] Figure 3 This is a schematic diagram showing the location of the center point of the pupil in one embodiment;

[0054] Figure 4 This is a schematic diagram illustrating the principle of determining the center point of the pupil in one embodiment;

[0055] Figure 5 This is a diagram illustrating the application environment of a head-mounted control module in one embodiment;

[0056] Figure 6 This is a schematic diagram of the structure of the processing component in one embodiment;

[0057] Figure 7 An exploded view of a head-mounted device in one embodiment;

[0058] Figure 8 This is a schematic diagram of the surgical instrument in one embodiment;

[0059] Figure 9 This is a schematic diagram of the surgical instruments moving to the right in one embodiment;

[0060] Figure 10 This is a schematic diagram of the surgical instruments moving to the left in one embodiment;

[0061] Figure 11 This is an internal structure diagram of a head-mounted control module in one embodiment;

[0062] Figure 12 This is a block diagram of the surgical instrument control system in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0065] In one embodiment, such as Figure 1 As shown, this application also provides a surgical instrument control method, which can be applied to a head-mounted control module, including:

[0066] S101: Acquire eye image data and head motion data.

[0067] Among them, eye image data and head motion data can be collected by the head-mounted control module.

[0068] S102: Obtain the current control mode of the head-mounted control module.

[0069] Specifically, the head-mounted control module has multiple control modes, such as an eye-movement-based control mode, a head-movement-based control mode, and a combined control mode. When not in the corresponding control mode, neither eye movement nor head movement affects the movement of the surgical instruments.

[0070] S103: Obtain target control data according to the current control mode, and control the lens movement of the endoscope according to the target control data, wherein the target control data includes at least one of eye movement data and head movement data, and the eye movement data is obtained based on eye image data.

[0071] In this embodiment, after receiving the user's eye image data and head movement data, the current control mode is obtained. Target control data is then acquired based on the current control mode, and the endoscope's lens movement is controlled according to the target control data, thereby achieving endoscope pose adjustment. The acquisition of eye image data and head movement data relies on the user's eye and / or head movements, eliminating the need for hand and foot coordination, thus simplifying endoscope control and making operation more convenient for the user. Furthermore, target control data is only acquired when the corresponding control mode is entered. When not in a control mode, the user's eye and head movements do not issue movement commands. By setting control modes, the probability of erroneous movement commands can be reduced. The target control data includes at least one of eye movement data and head movement data. Since eye movements are relatively rapid and head movements are relatively stable and reliable, each control mode has different control characteristics. The control mode can be selected according to the application scenario to meet the usage requirements of various application scenarios.

[0072] In one embodiment, obtaining eye image data includes the steps of: obtaining a user's face image and inputting the face image into a trained eye detection model to obtain the user's eye image data.

[0073] The eye detection model is obtained through the following steps:

[0074] Obtain a second sample set, which includes sample face images and corresponding training labels. The training labels are rectangular boxes that enclose the positions of the eyeballs. Input the sample face images into the object detection model and output a predicted image with a prediction box. Determine the loss function based on the predicted image and the training labels. Train the object detection model based on the training labels and the loss function to obtain the eye detection model.

[0075] For example, the object detection model can be based on the YOLOv4 algorithm. The specific training process is as follows: First, collect n (e.g., 2000) sets of various different sample face images. Use rectangular boxes to frame the position of the eyeballs to create training labels for each image, preparing for training the YOLOv4 object detection model. During training, the labeled sample set is placed into the YOLOv4 object detection model according to, for example, an image size of 640*640 pixels. The YOLOv4 object detection model uses CIOU to calculate the loss during training, and its calculation formula is as follows:

[0076] LOSS_CIOU=1-IOU+(ρ 2 *(b,b gt ) / c 2 )+αv (1)

[0077] The ratio of the intersection to the union of the predicted bounding box and the ground truth bounding box is defined as IOU. We use ρ 2 *(b,b gt ) is used to calculate the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes; the expansion formulas for α and v in the formula are as follows:

[0078] α=v / (1-IOU+v) (2)

[0079] v=(4 / π 2 )*(arctan(w gt / h gt )-arctan(w / h)) 2 (3)

[0080] w represents the width; h represents the height; gt represents the ground truth data label.

[0081] Using the trained YOLOv4 object detection model, we obtain eye (x,y) data by utilizing the model's output and NMS filtering, i.e., we acquire eye image data.

[0082] In one embodiment, the surgical instrument control method further includes the steps of: calculating grayscale values ​​of eye image data, determining coordinate data of the eyeball center, and determining eyeball movement data based on the coordinate data of the eyeball center.

[0083] Specifically, such as Figure 2As shown, after acquiring the eye image data, the corresponding regions of interest (ROIs) 31 for the left and right eyes are obtained. The eye ROI images are determined based on human eye proportions and integral calculations. Then, according to grayscale calculation methods, the closer to the center of the eyeball, the lower the grayscale value, allowing for the extraction of the pupil contour. Finally, the pupil edge pixels, approximately circular or elliptical in shape, are obtained. Finally, as... Figure 3 As shown, the center point is calculated using a circle or ellipse, which is the center point 32 of the pupil, thus enabling the positioning and tracking of the eyeball.

[0084] Among them, such as Figure 4 As shown, the pupil center point can be calculated as follows: After obtaining the regions of interest for the left and right eyes, the circumscribed rectangle ABCD of the pupil is established, thus obtaining the tangent points P1(x1, y1), P2(x2, y2), P3(x3, y3), and P4(x4, y4) between the pupil outline and its circumscribed rectangle. Then, based on each tangent point, the position coordinates P(x0, y0) of the center point of symmetry of the circumscribed rectangle can be obtained, calculated using the following formula:

[0085] x0 = (x1 - x3) / 2; (4)

[0086] y0 = (y2 - y4) / 2; (5)

[0087] In one embodiment, eye movement data includes pupil center coordinates and three-dimensional matrix data representing head posture. The process of acquiring target control data based on the current control mode and controlling the movement of surgical instruments based on the target control data includes: when the current control mode is a first integrated control mode, multiplying the user's current pupil center coordinates with the three-dimensional matrix data representing the current head posture after expansion to obtain a target one-dimensional matrix; inputting the target one-dimensional matrix into a trained motion control model, and controlling the movement of surgical instruments based on the output data of the motion control model.

[0088] The motion control model is obtained through the following steps: acquiring a first sample set, which includes a one-dimensional sample matrix and training labels corresponding to the one-dimensional sample matrix representing the direction of movement of the surgical instrument. The one-dimensional sample matrix is ​​obtained by multiplying the pupil center coordinates of the sample matrix with the three-dimensional sample matrix data; inputting the one-dimensional sample matrix into a neural network model, which outputs the three-dimensional direction values ​​corresponding to the one-dimensional sample matrix; and training the neural network model based on the label values ​​and the three-dimensional direction values ​​to obtain the motion control model.

[0089] Specifically, the coordinate values ​​and the three-dimensional matrix output by the head sensor are expanded and multiplied to obtain a one-dimensional matrix, which is used as the input to the neural network model. The neural network model finally outputs three-dimensional direction values ​​(x, y, z).

[0090] For example, the training process of the neural network model can be as follows: 1000 sets of eye positioning data and head sensor data are collected respectively. Then, the direction of movement of the surgical instrument as reflected by these data is used as the label value (x_label, y_label, z_label) as the training label. Finally, the model training inputs the data into a layer containing 256 neurons. The mathematical representation of the output of a single neuron is as follows:

[0091]

[0092] In formula (6), f is the transfer function, w is the weight vector used for weight adjustment during training, A is the input vector which is the one-dimensional matrix resulting from the multiplication of eye positioning data and head sensor data, and b is the bias matrix.

[0093] In this process, the silu function is used to extract key information, as shown in the following formula:

[0094] f(x)=x*sigmoid(x) (7)

[0095] Where x is the output of the neural network described above, and the expansion formula of the sigmoid function is as follows:

[0096]

[0097] In formula (8), e is a natural constant. The output of each neuron is normalized by using the output range of the sigmoid function from 0 to 1.

[0098] Finally, the target classification is achieved by using the Softmax function as the output layer;

[0099] The Softmax function is as follows:

[0100]

[0101] In formula (9), C is the final number of categories, e is a natural constant, and the final zi is the output value of the final movement direction of the surgical instrument.

[0102] In one embodiment, the surgical instrument control method further includes: when the current control mode is eye control mode, obtaining changes in the center position of the user's pupil based on the user's eye image data to obtain pupil data, obtaining eye movement data based on the pupil data, and controlling the lens posture of the endoscope based on the eye movement data, wherein the eye movement data is target control data.

[0103] The direction of rotation of the user's pupil center position can be used to adjust the endoscope lens. For example, if the user's pupil center position rotates to the left, the endoscope lens will shift to the left.

[0104] After acquiring eye image data, the center point of the pupil can be determined using the grayscale value calculation method described above, and then the lens posture of the endoscope can be controlled based on the position change of the pupil center point.

[0105] In this embodiment, in eye control mode, the user can control the lens posture of the endoscope by rotating their eyes. Since the eye movements are rapid, the user can quickly observe the surrounding situation of the surgical area during the operation, which helps to improve the efficiency of the operation.

[0106] In one embodiment, the surgical instrument control method further includes: when the current control mode is head control mode, acquiring head motion data, and controlling the lens movement of the endoscope according to the head motion data, wherein the head motion data is target control data.

[0107] In head control mode, users can control the movement of the endoscope lens by moving their head. The head movement is highly controllable, relatively stable and reliable, and the endoscope lens movement can be adjusted more accurately and stably based on the head movement.

[0108] Taking an endoscope as an example, the endoscope lens movement commands include position commands and posture commands. The head control modes include position adjustment mode and posture adjustment mode. The head movement data includes head translation data and head rotation data. When the current control mode is position adjustment mode, the endoscope lens translation movement is controlled according to the head translation data (i.e., the endoscope lens is adjusted). When the current control mode is posture adjustment mode, the endoscope lens posture is controlled according to the head rotation data (i.e., the endoscope lens posture is adjusted).

[0109] In one embodiment, the surgical instrument control method further includes: when the current control mode is a posture adjustment mode, acquiring the rotation direction, rotation angle and action holding time of the user's current posture relative to the reference posture, determining whether the user's current posture is a valid control posture based on the action holding time, and outputting a posture command based on the rotation direction and rotation angle when the current posture is a valid control posture.

[0110] The reference posture refers to the user's normal posture, such as a posture where the head does not rotate and the user can maintain it for an extended period. For example, the rotation direction of the user's current posture relative to the reference posture can be used as the rotation direction of the endoscope lens. The rotation angle of the user's current posture relative to the reference posture is proportional to the rotation angle of the endoscope lens, thereby achieving precise control of the endoscope lens.

[0111] In this embodiment, the user's current posture is determined as a valid control posture by measuring the duration of the action. This avoids frequent head movements that could cause frequent endoscope movement and reduces the probability of erroneous movement commands. The reference posture is the user's normal posture, which is relatively comfortable. After the user controls the endoscope movement by head movements, they will remain in an abnormal posture, which is uncomfortable for a long time. Since the posture command is output based on the rotation direction and angle relative to the reference posture, it can be understood that the endoscope will not move when the user is in the reference posture. Therefore, based on the solution in this embodiment, after the endoscope moves to a designated position, the user can quickly return to the reference posture, and the endoscope will remain in that designated posture.

[0112] In addition, the surgical instrument control method also includes: when the current control mode is the second comprehensive control mode, obtaining the current eye rotation direction of the user based on eye movement data, obtaining the head rotation direction, head rotation angle and action holding time of the user's head relative to the reference posture based on head movement data, determining whether the user's current posture is an effective control posture based on the action holding time, and determining the rotation direction of the surgical instrument based on the eye rotation direction and head rotation direction when the current posture is an effective control posture, and determining the rotation angle of the surgical instrument based on the head rotation angle.

[0113] The eyeball has a wide range of motion, while the head has a narrower range of motion. Combining the direction of eyeball rotation and the direction of head rotation can expand the control range of the rotation direction. When the deviation angle between the head rotation direction and the eyeball rotation direction does not exceed a preset range, the direction of eyeball rotation is determined as the rotation direction of the surgical instrument. For example, when the eyeball rotates to the upper left and the head rotates to the left, the deviation angle between the eyeball and the head is less than 90 degrees. Therefore, the direction of eyeball rotation is determined as the rotation direction of the surgical instrument, and the rotation direction of the surgical instrument is upper left.

[0114] In one embodiment, a head-mounted control module is provided, the internal structure of which can be shown in the following diagram. Figure 11As shown, the head-mounted control module includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media. The database stores eye image data and head movement data. The I / O interfaces allow the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a surgical instrument control method.

[0115] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the head-mounted control module to which the present application is applied. A specific head-mounted control module may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] Based on the above embodiments, in one embodiment, this application provides a head-mounted control module, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the surgical instrument control method as described above. Specifically, as... Figures 5 to 7 As shown, the head-mounted control module includes a head-mounted device 11 and a processing component 12 as an example for explanation.

[0117] The head-mounted device 11 includes a housing 112, a data acquisition module 111, and a first communication module 113. The data acquisition module 111 and the first communication module 113 are located inside the housing 112. The data acquisition module 111 is connected to the first communication module 113. The data acquisition module 111 is used to acquire the user's eye image data and head movement data.

[0118] The processing component 12 includes a processing module 121 and a second communication module 122. The second communication module 122 is communicatively connected to the processing module 121, the first communication module 113, and the surgical instruments. The second communication module 122 is used to receive eye image data and head motion data sent by the first communication module 113. The processing module 121 is used to obtain the current control mode, obtain target control data according to the current control mode, generate motion commands according to the target control data, and control the second communication module 122 to send the motion commands to the surgical instruments (this application uses endoscope 21 as an example) to control the movement of the surgical instruments. The target control data includes at least one of eye movement data and head motion data. The eye movement data is obtained by the processing component 12 based on the eye image data.

[0119] The head-mounted device 11 is worn on the user's head and covers the user's eyes to facilitate the acquisition module 111 in collecting image data of the user's eyes. The head-mounted device 11 can be a virtual reality device. It can be understood that after the head-mounted device 11 is worn on the user's head, it will move synchronously with the user's head. Based on this, the movement of the head-mounted device 11 can be collected, thereby collecting head movement data. Figure 2 As shown, the processing component 12 may also include a heat dissipation module, such as a fan, to dissipate heat from the processing module 121 and the second communication module 122. Surgical instruments may include instruments such as energy devices and endoscopes 21; this application uses endoscopes 21 as an example for description.

[0120] In applications, such as Figure 5 As shown, the head-mounted device 11 and the processing component 12 can be separated, thereby reducing the weight of the head-mounted device 11 and improving the user experience when it is worn on the user's head. Optionally, the processing component 12 can also be located within the housing 112, making the head-mounted control module an integrated structure and improving its portability. The first communication module 113 and the second communication module 122 can communicate wirelessly or via a wired connection, depending on the actual application scenario. If portability is prioritized, the first communication module 113 and the second communication module 122 can communicate wirelessly, for example, via Bluetooth technology; if data transmission stability is prioritized, the first communication module 113 and the second communication module 122 can communicate via a wired connection.

[0121] It is understandable that when the head-mounted control module is not in the corresponding control mode, the processing component 12 will not issue corresponding motion commands based on the user's eye and head movements. This allows the user's eyes and head to move freely, avoiding the issuance of erroneous motion commands. However, when the head-mounted control module enters the corresponding control mode, the processing module 121 can only generate motion commands based on the target control data corresponding to that control mode, thereby reducing the probability of erroneous motion commands.

[0122] The aforementioned head-mounted control module includes a head-mounted device 11 and a processing component 12. The head-mounted device 11 collects the user's eye image data and head movement data and sends them to the processing component 12. The processing component 12 determines the target control data corresponding to the current control mode of the head-mounted control module, and controls the movement of the lens 214 of the endoscope 21 according to the target control data, thereby realizing the position and posture adjustment of the lens 214 of the endoscope 21. Since the user only needs to move their eyes and / or head, the position and posture of surgical instruments can be controlled by the doctor's foot pedal and main hand, as in related technologies. Therefore, the control operation of the endoscope 21 is simplified, and the user operation is more convenient. In addition, since eye movements are relatively rapid and head movements are relatively stable and reliable, each control mode has different control characteristics. The control mode can be selected according to the application scenario to meet the usage requirements of various application scenarios.

[0123] In one embodiment, the head-mounted device 11 further includes a display screen located inside the housing 112 and connected to the first communication module 113; the processing module 121 is also used to receive image data collected by the endoscope 21 and control the second communication module 122 to send the image data to the first communication module 113 so as to display the corresponding image on the display screen.

[0124] In this embodiment, after the endoscope 21 acquires an image of the area in front of the lens 214, the processing component 12 sends the image data acquired by the endoscope 21 to the head-mounted device 11 so that the image acquired by the endoscope 21 can be displayed on the display screen of the head-mounted device 11, thereby increasing the user's sense of immersion and improving the accuracy and efficiency of the surgery. After the image acquired by the endoscope 21 is displayed on the display screen of the head-mounted device 11, the user can determine the adjustment method of the endoscope 21 based on the displayed image, control the head-mounted device 11 to enter the corresponding control mode, and then control the movement of the lens 214 of the endoscope 21 to observe the surgical site and the surrounding conditions.

[0125] In one embodiment, the acquisition module 111 includes an image acquisition unit and a motion sensing unit. The image acquisition unit is connected to the first communication module 113 and is used to acquire eye image data of the user; the motion sensing unit is connected to the first communication module 113 and is used to acquire head movement data of the user.

[0126] The image acquisition unit may include a camera, and the motion sensing unit may include a posture sensor. The image acquisition range of the image acquisition unit includes the user's left and right eyes.

[0127] In one embodiment, the image acquisition unit includes an infrared camera used to acquire image data of the user's eyes.

[0128] The infrared camera can be two, one facing the user's left eye and the other facing their right eye.

[0129] In this embodiment, the user's eye image data is collected by an infrared camera. Based on the characteristics of infrared cameras, no light source is required, and the user's eye image data can be collected in dark environments. Moreover, the response speed is fast, which helps to improve the acquisition efficiency.

[0130] Of course, you can also use an infrared camera to collect the user's facial image data, and then input the facial image into a trained eye detection model to obtain the user's eye image data.

[0131] In one embodiment, the head-mounted device 11 further includes a mode switching module, which is used to receive mode control commands input by the user, and the processing module 121 is also used to control the head-mounted control module to enter or exit the corresponding control mode according to the mode control commands.

[0132] Optionally, the mode switching module also includes a voice acquisition module, which is used to receive voice information, and the processing module 121 is also used to determine whether to enter or exit the corresponding control mode based on the voice information.

[0133] The voice acquisition module may include a microphone, which is housed within the housing 112. After acquiring voice information, the voice acquisition component transmits the voice information to the second communication module 122 via the first communication module 113. The second communication module 122 then transmits the voice information to the processing module 121. The processing module 121 analyzes the voice information to determine if there are any keywords corresponding to a preset control mode. If a corresponding keyword exists, the head-mounted control module is controlled to enter the corresponding control mode; otherwise, no response is made. For example, the keyword corresponding to control mode A is "enter control mode A". If the keyword "enter control mode A" is identified from the voice information, the head-mounted control module is controlled to enter control mode A.

[0134] It is understandable that, in addition to entering or exiting the corresponding control mode via voice control, buttons for the corresponding control modes can also be set on the head-mounted device 11, and the corresponding control mode can be entered or exited by pressing the buttons.

[0135] In applications, the movement of the endoscope 21 can also be controlled by voice. For example, the control mode includes a voice control mode. When the voice control mode is entered, the processing module 121 identifies whether there are corresponding keywords related to the movement of the endoscope 21 in the voice information. If the corresponding keywords are present, the module outputs the movement command corresponding to the keywords to control the movement of the endoscope 21. It is understood that the image displayed on the screen can also be controlled by voice, for example, by controlling the zoom in and out of the displayed image.

[0136] In one embodiment, the head-mounted device 11 may further include a voice prompt module for outputting voice prompt information when the head-mounted control module enters or exits a corresponding control mode.

[0137] The voice prompt module may include a speaker. The voice prompt information is a prompt statement corresponding to the control mode. For example, when the head-mounted control module enters control mode A, the head-mounted device 11 outputs the prompt voice "Entered control mode A".

[0138] In conjunction with the surgical instrument control method described in the above embodiments, the control mode of the head-mounted control module may include a first comprehensive control mode. When the current control mode is the first comprehensive control mode, the processing module 121 expands the user's current pupil center coordinate value and multiplies it with the three-dimensional matrix data representing the current head posture to obtain a target one-dimensional matrix. The target one-dimensional matrix is ​​input into the trained motion control model, and the motion command of the surgical instrument is determined according to the output data of the motion control model. The motion command is sent to the surgical instrument through the second communication module 122, thereby controlling the movement of the surgical instrument.

[0139] Based on the above embodiments, the control mode of the head-mounted control module may also include an eye control mode. When the current control mode is the eye control mode, the processing module 121 obtains the change in the center position of the user's pupil based on the user's eye image data to obtain pupil data, obtains the user's eye movement data based on the pupil data, and outputs motion commands based on the eye movement data to instruct the posture of the lens 214 of the endoscope 21.

[0140] In this embodiment, in eye control mode, the user can control the posture of the lens 214 of the endoscope 21 by rotating the eyeball. Since the eyeball rotation is rapid, the user can quickly observe the situation around the surgical area by rotating the eyeball during the operation, which helps to improve the efficiency of the operation.

[0141] In one embodiment, the control mode of the head-mounted control module may also include an eye-control mode. When the current control mode is head control mode, head motion data is acquired, and motion commands are output based on the head motion data to instruct the movement of the lens 214 of the endoscope 21. The head motion data is the target control data.

[0142] In this embodiment, when the current control mode is head control mode, the user can control the movement of the lens 214 of the endoscope 21 by moving their head. The head movement is highly controllable, relatively stable and reliable, so the movement of the lens 214 of the endoscope 21 can be adjusted more accurately and stably based on the head movement.

[0143] In one embodiment, the motion commands include position commands and posture commands, the head control modes include position adjustment modes and posture adjustment modes, and the head motion data includes head translation data and head rotation data. The processing module 121 is further configured to, when the current control mode is position adjustment mode, acquire head translation data of the head motion data, and output position commands based on the head translation data to instruct the lens 214 of the endoscope 21 to translate; and when the current control mode is posture adjustment mode, output posture commands based on the head rotation data to instruct the posture of the lens 214 of the endoscope 21.

[0144] In this embodiment, by setting a position adjustment mode and a posture adjustment mode, the position and posture of the endoscope 21 lens 214 can be controlled by head movement. Since the position adjustment mode and the posture adjustment mode are two modes, the adjustment process of the position and posture of the endoscope 21 lens 214 will not interfere with each other, thereby adjusting the position and posture of the endoscope 21 lens 214 more accurately.

[0145] In one embodiment, the processing module 121 is further configured to, when the current control mode is the attitude adjustment mode, obtain the rotation direction, rotation angle and action holding time of the user's current attitude relative to the reference attitude, determine whether the user's current attitude is a valid control attitude based on the action holding time, and output attitude commands based on the rotation direction and rotation angle when the current attitude is a valid control attitude.

[0146] In this embodiment, the user's current posture is determined by the duration of the action to ensure it is a valid control posture. This avoids frequent head movements causing frequent movement of the endoscope 21's lens 214 and reduces the probability of erroneous movement commands. The reference posture is the user's normal posture, which is comfortable for the user. After the user controls the endoscope 21's lens 214 to move through head movements, it remains in an abnormal posture. Maintaining this abnormal posture for a long time is uncomfortable. Since the posture command is output based on the rotation direction and angle relative to the reference posture, it can be understood that the endoscope 21's lens 214 will not move when the user is in the reference posture. Therefore, based on the solution of this embodiment, after the endoscope 21's lens 214 moves to the designated position, the user can quickly return to the reference posture, and the endoscope 21's lens 214 will remain in that designated posture.

[0147] In one embodiment, the processing module 121 is further configured to, when the current control mode is the second comprehensive control mode, obtain the current eye rotation direction of the user based on eye movement data, obtain the head rotation direction, head rotation angle and action holding time of the user's head relative to the reference posture based on head movement data, determine whether the user's current posture is a valid control posture based on the action holding time, and when the current posture is a valid control posture, determine the rotation direction of the surgical instrument based on the eye rotation direction and head rotation direction, and determine the rotation angle of the surgical instrument based on the head rotation angle.

[0148] Based on the same inventive concept, taking an endoscope as a surgical instrument as an example, such as Figures 5 to 8 As shown, in one embodiment, this application provides an endoscope control system, which includes an endoscope 21 and a head-mounted control module as described in any of the above embodiments. The endoscope 21 includes a drive module 212 and a third communication module 211. The third communication module 211 is connected to both the drive module 212 and a second communication module 122. The third communication module 211 is used to receive motion commands, and the drive module 212 is used to drive the lens 214 of the endoscope 21 to move according to the motion commands.

[0149] The endoscope 21 is a flexible endoscope 21, and its posture can be adjusted. Specifically, for example... Figure 4 As shown, the endoscope 21 may include a drive module 212, a third communication module 211, a flexible rod 213, and a lens 214. The flexible rod 213 is connected to both the drive module 212 and the lens 214. After receiving a motion command, the drive module 212 drives the flexible rod 213 to bend and / or extend / retract according to the motion command, thereby driving the lens 214 to move. The endoscope's offset process can be as follows: Figure 9 and Figure 10 As shown, Figure 9 This refers to the process of moving the endoscope to the right. Figure 10 This refers to the process of moving the endoscope to the left.

[0150] The aforementioned endoscope control system collects the user's eye image data and head movement data through the head-mounted device 11 and sends them to the processing component 12. The processing component 12 obtains target control data according to the current control mode, generates motion commands based on the target control data, and sends them to the third communication module 211 through the second communication module 122. The endoscope 21 controls the movement of the lens 214 of the endoscope 21 based on the motion commands, thereby realizing the adjustment of the position and posture of the endoscope 21. The user's adjustment process of the endoscope 21 does not require hand and foot coordination, but only requires the user's eyeballs and / or head movements. Therefore, the control operation of the endoscope 21 is simplified, and the user operation is more convenient. Furthermore, only after the head-mounted device 11 enters the corresponding control mode can the user issue movement commands through eye movements and / or head movements. When the head-mounted device 11 is not in a control mode, the user's eye movements and head movements will not issue movement commands. By setting the control mode, the probability of erroneous movement commands can be reduced. The target control data includes at least one of eye movement data and head movement data. Since eye movements are relatively rapid and head movements are relatively stable and reliable, each control mode has different control characteristics. The control mode can be selected according to the application scenario to meet the usage requirements of each application scenario.

[0151] In one embodiment, such as Figure 12 As shown, this application also provides a surgical instrument control system 120, including: a first acquisition module 1201, a second acquisition module 1202 and a control module 1203.

[0152] The first acquisition module 1201 is used to acquire eye image data and head motion data.

[0153] The second acquisition module 1202 is used to acquire the current control mode of the head-mounted control module.

[0154] The control module 1203 is used to acquire target control data according to the current control mode and control the movement of surgical instruments according to the target control data. The target control data includes at least one of eye movement data and head movement data, and the eye movement data is acquired based on eye image data.

[0155] In one embodiment, the eye movement data includes pupil center coordinates and three-dimensional matrix data representing head posture. The control module 1203 includes a product submodule and a first control submodule. The product submodule is used to multiply the user's current pupil center coordinates with the three-dimensional matrix data representing the current head posture after expansion when the current control mode is the first comprehensive control mode to obtain a target one-dimensional matrix. The control submodule is used to input the target one-dimensional matrix into a trained motion control model and control the movement of surgical instruments according to the output data of the motion control model.

[0156] The motion control model is obtained through the following steps:

[0157] Obtain a first sample set, which includes a one-dimensional sample matrix and training labels corresponding to the one-dimensional sample matrix representing the direction of movement of the surgical instruments. The one-dimensional sample matrix is ​​obtained by multiplying the pupil center coordinates of the sample with the three-dimensional sample matrix data. Input the one-dimensional sample matrix into the neural network model and output the three-dimensional direction values ​​corresponding to the one-dimensional sample matrix. Train the neural network model based on the label values ​​and the three-dimensional direction values ​​to obtain the motion control model.

[0158] In one embodiment, the control module 1203 further includes a first acquisition submodule, which is used to acquire the user's facial image and input the facial image into a trained eye detection model to obtain the user's eye image data.

[0159] The eye detection model is obtained through the following steps: acquiring a second sample set, which includes sample face images and corresponding training labels, where the training labels are rectangular boxes that enclose the positions of the eyeballs; inputting the sample face images into the object detection model and outputting a predicted image with a prediction box; determining a loss function based on the predicted image and the training labels; and training the object detection model based on the training labels and the loss function to obtain the eye detection model.

[0160] In one embodiment, the surgical instrument control system 120 further includes a calculation module, which is used to calculate grayscale values ​​of eye image data, determine the coordinate data of the eyeball center, and determine eyeball movement data based on the coordinate data of the eyeball center.

[0161] In one embodiment, the control module 1203 further includes: a second acquisition submodule, a third acquisition submodule, a judgment submodule, and a determination submodule. The second acquisition submodule is used to acquire the current user's eye rotation direction based on eye movement data when the current control mode is the second comprehensive control mode. The third acquisition submodule is used to acquire the user's head rotation direction, head rotation angle, and action holding time relative to the reference posture based on head movement data. The judgment submodule is used to determine whether the user's current posture is a valid control posture based on the action holding time. The determination submodule is used to determine the rotation direction of the surgical instrument based on the eye rotation direction and head rotation direction when the current posture is a valid control posture, and to determine the rotation angle of the surgical instrument based on the head rotation angle.

[0162] In one embodiment, the control module 1203 further includes: a fourth submodule, a fifth submodule, and a second control submodule. The fourth submodule is used to obtain pupil data by acquiring changes in the center position of the user's pupil based on the user's eye image data when the current control mode is eye control mode. The fifth submodule is used to acquire eye movement data based on the pupil data. The second control submodule is used to control the movement of surgical instruments based on the eye movement data, wherein the eye movement data is target control data.

[0163] In one embodiment, the control module 1203 further includes a sixth acquisition submodule and a third control submodule. The sixth acquisition submodule is used to acquire head motion data when the current control mode is head control mode. The third control submodule is used to control the movement of surgical instruments based on the head motion data, wherein the head motion data is target control data.

[0164] In one embodiment, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the surgical instrument control method as described in any of the above embodiments.

[0165] In one embodiment, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the surgical instrument control method as described in any of the above embodiments.

[0166] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Furthermore, any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0167] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for controlling surgical instruments, characterized in that, Applications include head-mounted control modules; The surgical instrument control method includes: Acquire eye image data and head motion data; Obtain the current control mode of the head-mounted control module; Target control data is acquired according to the current control mode, and the movement of surgical instruments is controlled according to the target control data. The target control data includes at least one of eye movement data and head movement data, and the eye movement data is acquired based on the eye image data. The eye movement data includes pupil center coordinates and three-dimensional matrix data representing head posture; acquiring target control data based on the current control mode and controlling the movement of surgical instruments based on the target control data includes: When the current control mode is the first comprehensive control mode, the user's current pupil center coordinates are expanded and multiplied with the three-dimensional matrix data representing the current head posture to obtain the target one-dimensional matrix; The target one-dimensional matrix is ​​input into the trained motion control model, and the movement of the surgical instrument is controlled according to the output data of the motion control model.

2. The surgical instrument control method according to claim 1, characterized in that, The motion control model is obtained through the following steps: Obtain a first sample set, which includes a one-dimensional sample matrix and training labels corresponding to the one-dimensional sample matrix that characterize the direction of movement of surgical instruments. The one-dimensional sample matrix is ​​obtained by multiplying the pupil center coordinates of the sample with the three-dimensional sample matrix data. The one-dimensional sample matrix is ​​input into the neural network model, and the three-dimensional direction value corresponding to the one-dimensional sample matrix is ​​output. The neural network model is trained based on the label value and the three-dimensional direction value to obtain the motion control model.

3. The surgical instrument control method according to claim 1, characterized in that, The acquisition of eye image data includes: Acquire the user's facial image and input the facial image into a trained eye detection model to obtain the user's eye image data; The eye detection model is obtained through the following steps: Obtain a second sample set, which includes sample face images and training labels corresponding to the sample face images. The training labels are rectangular boxes that frame the position of the eyeballs. The sample face image is input into the target detection model, and the output is a predicted image with a prediction box; The loss function is determined based on the predicted image and the training labels; The target detection model is trained based on the training labels and the loss function to obtain the eye detection model.

4. The surgical instrument control method according to any one of claims 1 to 3, characterized in that, The surgical instrument control method further includes: The grayscale value of the eye image data is calculated to determine the coordinate data of the eyeball center, and the eyeball movement data is determined based on the coordinate data of the eyeball center.

5. The surgical instrument control method according to claim 1, characterized in that, The step of acquiring target control data according to the current control mode and controlling the movement of surgical instruments according to the target control data includes: When the current control mode is the second comprehensive control mode, the current user's eye rotation direction is obtained based on the eye movement data; Based on the head motion data, the user's head rotation direction, head rotation angle, and motion holding time relative to the reference posture are obtained; Determine whether the user's current posture is a valid control posture based on the duration of the action; When the current posture is an effective control posture, the rotation direction of the surgical instrument is determined according to the rotation direction of the eyeball and the rotation direction of the head, and the rotation angle of the surgical instrument is determined according to the rotation angle of the head.

6. The surgical instrument control method according to claim 1, characterized in that, The step of acquiring target control data according to the current control mode and controlling the movement of surgical instruments according to the target control data includes: When the current control mode is eye control mode, the change in the center position of the user's pupil is obtained based on the user's eye image data to obtain pupil data; The eye movement data is obtained based on the pupil data; The movement of the surgical instrument is controlled based on the eye movement data, wherein the eye movement data is the target control data.

7. The surgical instrument control method according to claim 1, characterized in that, The step of acquiring target control data according to the current control mode and controlling the movement of surgical instruments according to the target control data includes: When the current control mode is head control mode, the head motion data is acquired; The movement of the surgical instrument is controlled based on the head movement data, wherein the head movement data is the target control data.

8. The surgical instrument control method according to claim 7, characterized in that, The motion commands include position commands and posture commands; the head control modes include position adjustment modes and posture adjustment modes; and the head motion data includes head translation data and head rotation data. The surgical instrument control method further includes: When the current control mode is position adjustment mode, head translation data of the head motion data is acquired; The surgical instruments are controlled to perform translational movements based on the head translation data. When the current control mode is posture adjustment mode, the posture of the surgical instrument is controlled according to the head rotation data.

9. The surgical instrument control method according to claim 7, characterized in that, The surgical instrument control method further includes: When the current control mode is attitude adjustment mode, obtain the user's current attitude relative to the reference attitude, rotation direction, rotation angle and action holding time; Determine whether the user's current posture is a valid control posture based on the duration of the action; When the current posture is an effective control posture, the posture command is output according to the rotation direction and rotation angle.

10. A surgical instrument control system, characterized in that, include: The first acquisition module is used to acquire eye image data and head motion data; The second acquisition module is used to acquire the current control mode of the head-mounted control module; A control module is configured to acquire target control data according to the current control mode, and control the movement of surgical instruments according to the target control data, wherein the target control data includes at least one of eye movement data and head movement data, and the eye movement data is acquired based on the eye image data; The eye movement data includes pupil center coordinates and three-dimensional matrix data representing head posture. The control module is also used to, when the current control mode is the first comprehensive control mode, expand the user's current pupil center coordinates and multiply them with the three-dimensional matrix data representing the current head posture to obtain a target one-dimensional matrix. The target one-dimensional matrix is ​​input into the trained motion control model, and the movement of the surgical instrument is controlled according to the output data of the motion control model.

11. A head-mounted control module, characterized in that, The method includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images