A fully automatic robot-assisted nasal endoscopy method and system

Abstract

The present invention provides a fully automatic robot-assisted nasal endoscopy method and system, including a nasal endoscope, an image preprocessing module, an optical positioning module, a motion planning module, a force sensing module, and a control module. The fully automatic nasal endoscopy method includes the following steps: the robot clamps the nasal endoscope to an initial position; obtains an endoscopic image, and preprocesses the endoscopic image for image enhancement and optical flow estimation; inputs the endoscopic image, the corresponding optical flow map, and the current state of the robot into a deep learning network for motion and path planning; monitors the contact between the endoscope and the nasal tissue based on a force sensor system provided on the endoscope; and controls the robot's motion based on motion planning results and force feedback data. The present invention adopts the above-mentioned fully automatic robot-assisted nasal endoscopy method and system to control the robot to fully automatically perform the nasal endoscopy inspection process, thereby improving the efficiency and stability of nasal endoscopic surgery and diagnosis and treatment.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and automatic control, and in particular to a fully automatic robot-assisted nasal endoscopy method and system. Background Art

[0002] Nasal endoscopy is a commonly used diagnostic and treatment method in otolaryngology, widely used in the diagnosis and treatment of diseases of the nasal cavity, sinuses, and related structures. Traditional nasal endoscopy relies primarily on a handheld endoscope, requiring the physician to manually control the angle and position of the endoscope to obtain the optimal field of view. This manual operation not only requires the physician to possess a high level of professional skills and extensive experience, but also can lead to fatigue from prolonged handheld operation, compromising the accuracy and efficiency of the examination. Furthermore, manual operation can lead to errors due to human error, increasing patient discomfort and even risk.

[0003] With the rapid development of robotics and image processing technologies, robotic-assisted nasal endoscopy systems have gradually become a research hotspot. Fully automated robotic-assisted nasal endoscopy systems offer significant advantages in practical applications. First, they significantly improve examination accuracy and stability, reducing errors caused by manual operation. Second, the introduction of robotic systems can effectively reduce physician workload and operational errors caused by fatigue. Furthermore, with the help of intelligent assistance systems, physicians can complete examinations and diagnoses more quickly and accurately, improving work efficiency and medical quality. Summary of the Invention

[0004] The purpose of the present invention is to provide a fully automatic robot-assisted nasal endoscopy method and system, which can control the robot to fully automatically perform the nasal endoscopy examination process, thereby improving the efficiency and stability of nasal endoscopic surgery and diagnosis and treatment.

[0005] To achieve the above object, the present invention provides a fully automatic robot-assisted nasal endoscopy method comprising

[0006] Nasal endoscope, used to obtain endoscopic images of the inside of the nasal cavity and input them into the image preprocessing module;

[0007] An image preprocessing module is connected to the nasal endoscope and is used to perform image enhancement and optical flow estimation on the endoscopic image; and is connected to the action planning module and is used to input the obtained enhanced endoscopic image and optical flow map into the action planning module;

[0008] The optical positioning module is connected to the nasal endoscope to obtain the spatial position of the nasal endoscope; it is connected to the seven-degree-of-freedom robotic arm to determine the initial target position;

[0009] The motion planning module is connected to the image preprocessing module and is used to obtain and use the preprocessed image data to predict the target posture transformation; it is connected to the control module and is used to transmit the target posture change data to determine the robot's motion plan;

[0010] A force sensing module is connected to the nasal endoscope to measure the force data between the endoscope and the nasal tissue; and is connected to the control module to transmit and process force feedback data for robot control;

[0011] The control module is connected to the motion planning module to obtain and process the planned target motion sequence; connected to the force sensing module to obtain and process force sensing data; and connected to the seven-degree-of-freedom robotic arm to control the robot to complete the posture transformation target safely and stably.

[0012] The seven-degree-of-freedom robotic arm is connected to the control module to acquire and execute actions, connected to the nasal endoscope to clamp the endoscope; and connected to the optical positioning module to determine the initial position of the robot.

[0013] Preferably, the image preprocessing module includes an image enhancement unit and an optical flow estimation unit;

[0014] An image enhancement unit, connected to the nasal endoscope, for improving the resolution of images acquired by the nasal endoscope;

[0015] The optical flow estimation unit is connected to the nasal endoscope and is used to obtain the optical flow map of the endoscope image at adjacent moments.

[0016] Preferably, the optical positioning module includes

[0017] The rigid frame is used to install all the modules of the system. The optical locator and the locator host connection are set on the rigid frame;

[0018] Among them, the optical locator is used to transmit and receive infrared signals, and a reference frame is set on the optical locator.

[0019] Reference frame, used to reflect infrared signals emitted by the optical positioning instrument;

[0020] The locator host is connected to the optical locator and is used for spatial coordinate calculation and transmission.

[0021] Preferably, the motion planning module extracts feature representations of the time dimension and the space dimension based on the enhanced endoscopic image and the optical flow map, and performs motion planning based on the feature representations of the time dimension and the space dimension to obtain the target posture. The motion planning module includes

[0022] A spatial feature extraction unit, connected to the image preprocessing module, is used to extract feature representations of the endoscopic image in the spatial dimension;

[0023] A temporal feature extraction unit, connected to the image preprocessing module, is used to extract feature representations on the time dimension contained in the endoscopic image sequence;

[0024] The pose encoding unit is connected to the spatial feature extraction unit and the temporal feature extraction unit to predict the pose transformation vector and fuse the spatial and temporal features to represent the planned target pose.

[0025] Preferably, the image enhancement unit uses a diffusion model to obtain two consecutive frames of endoscopic images I i and I i-1 Perform image enhancement operation, expressed as At the same time, the optical flow estimation unit uses the FlowNet network to generate the corresponding optical flow map based on the current endoscopic image and the endoscopic image at the previous moment, which is expressed as

[0026] Preferably, the spatial feature extraction unit utilizes the spatial dimension features of the current endoscopic image and optical flow map using a convolutional neural network based on an attention mechanism, specifically including:

[0027] The endoscopic image is fed into the pre-attention mechanism module F. Attention ResNet50 encoder F ResNet50 , output the corresponding spatial feature vector The process is described by the following formula:

[0028]

[0029] In the attention mechanism module, the enhanced image of size (H, W, C) Obtain an input image of size (C, H, W) by dimension transformation and an input image of size (W,C,H) Then, adaptive average pooling is used to process F based on the original image and the transformed image. Avg , global pooling processing F Max And the convolutional layer F Conv Calculate the corresponding weight matrix {A I ,A J ,A k}; Use the weight matrix to multiply the corresponding original image element by element, and then average the weighted sum to get the output image It is described by the following formula:

[0030]

[0031]

[0032] Among them, A I Represents the input image The weight of A J Represents the input image The weight of A K Represents the input image The weight of

[0033] At the same time, the optical flow map Input ResNet50 encoder and output the corresponding spatial feature vector The process is described by the following formula:

[0034]

[0035] Preferably, the temporal feature extraction unit extracts the temporal dimension features of the optical flow graph, including:

[0036] Input the optical flow map into the ConvLSTM network F ConvLSTM , output the corresponding time series feature vector And the state vector h of the network i-1 Update, described by the following formula:

[0037]

[0038]

[0039] Among them, h i Indicates the current state of the network.

[0040] Preferably, the temporal feature vector and the spatial feature vector are concatenated in the channel dimension and input into the encoder F based on the gated convolutional neural network. GatedCNN , output the predicted target pose vector It is described by the following formula:

[0041]

[0042] Preferably, the control module obtains the motion vector Q of each joint of the robotic arm based on the compliant control method. t , to perform real-time automatic control of the seven-degree-of-freedom robotic arm, which can be described by the following formula:

[0043]

[0044] Among them, S t Indicates the current state of the seven-degree-of-freedom robotic arm, F t Represents force feedback data.

[0045] A fully automated robot-assisted nasal endoscopy method comprises the following steps:

[0046] The control module is used to control the robot to clamp the nasal endoscope to the initial position;

[0047] Acquiring endoscopic image data of the interior of the nasal cavity;

[0048] The endoscopic image is enhanced by using an image enhancement unit provided on the image preprocessing module, and a corresponding optical flow map is generated based on the current endoscopic image and the endoscopic image at the previous moment by using an optical flow estimation unit provided on the image preprocessing module, thereby completing the preprocessing of the endoscopic image data;

[0049] The enhanced endoscopic image and optical flow map are input into the motion planning module, feature representations of the time dimension and the space dimension are extracted from the enhanced endoscopic image and the optical flow map, and motion planning is performed based on the feature representations of the time dimension and the space dimension;

[0050] Using a force sensing module through the endoscope to obtain force feedback data between the endoscope and nasal tissue;

[0051] After the automatic motion planning is completed by the motion planning module, the target pose vector predicted by the motion planning module and the force feedback data F of the endoscope and nasal tissue obtained by the force sensing module at each sampling time t are used. t , and the current state S of the seven-degree-of-freedom manipulator t The data are input together into the control module, which controls the robot to execute the motion plan based on the motion planning and force feedback data.

[0052] Therefore, the present invention adopts the above-mentioned fully automatic robot-assisted nasal endoscopy method and system, which can control the robot to fully automatically perform the nasal endoscopy inspection process, thereby improving the efficiency and stability of nasal endoscopic surgery and diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a fully automatic robot-assisted nasal endoscopy method of the present invention;

[0054] Figure 2 This is a structural diagram of a fully automatic robot-assisted nasal endoscopy system;

[0055] Figure 3 It is a structural diagram of the image preprocessing module;

[0056] Figure 4 This is a structural diagram of the action planning module;

[0057] Figure 5 Schematic diagram of the process of fully automated robot-assisted nasal endoscopy. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0059] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0060] Example 1

[0061] like Figure 1 、 Figure 2 and Figure 3 As shown, the present invention provides a fully automatic robot-assisted nasal endoscopy method, comprising the following steps:

[0062] Step 1: Control the robot to hold the nasal endoscope and reach the initial position;

[0063] The spatial position of the nasal entrance is determined based on the optical positioning system and the reference frame as the target position of the robot.

[0064] The optical positioning system determines the nasal entrance based on the infrared signal reflected by the reference frame, and transmits the coordinates to the control system to control the robot to clamp the nasal endoscope to the initial position.

[0065] Step 2: The nasal endoscope obtains the endoscopic image of the nasal cavity and inputs it into the image preprocessing module.

[0066] Step 3: Preprocess the endoscopic image data

[0067] The diffusion model is used to perform image denoising, and the optical flow estimation network is used to obtain the optical flow map of the current image and the previous image.

[0068] like Figure 3 As shown in the figure, the image preprocessing module performs image enhancement and optical flow estimation on the endoscopic image:

[0069] The image enhancement unit uses the diffusion model to obtain two consecutive frames of endoscopic images I i and I i-1 Perform image enhancement operation, expressed as At the same time, the optical flow estimation unit uses the FlowNet network based on the current endoscopic image I i and the endoscopic image I at the previous moment i-1 Generate the corresponding optical flow map, expressed as

[0070] Step 4: The enhanced endoscopic image and optical flow map are input to the action planning module. The action planning module extracts the feature representation of the time dimension and the spatial dimension from the enhanced endoscopic image and the optical flow map, and performs action planning based on the feature representation of the time dimension and the spatial dimension, such as Figure 4 As shown.

[0071] A convolutional neural network based on the attention mechanism is used to extract the spatial dimension features of the image, and a recurrent neural network is used to extract the sequence dimension features of the image.

[0072] (1) Extract the spatial dimension features of the current endoscopic image and optical flow map. Input the endoscopic image into the pre-attention mechanism module F Attention ResNet50 encoder F ResNet50 , output the corresponding spatial feature vector The process is described by the following formula:

[0073]

[0074] In the attention mechanism module, the augmented image of size (H, W, C) Obtain an input image of size (C, H, W) by dimension transformation and an input image of size (W,C,H) Then, adaptive average pooling is used to process F based on the original image and the transformed image. Avg , global pooling processing F Max And the convolutional layer F Conv Calculate the corresponding weight matrix {A I ,A J ,A K}; Use the weight matrix to multiply the corresponding original image element by element, and then average the weighted sum to get the output image It is described by the following formula:

[0075]

[0076]

[0077] Among them, A I Represents the input image The weight of A J Represents the input image The weight of A K Represents the input image The weight of

[0078] At the same time, the optical flow map Input ResNet50 encoder and output the corresponding spatial feature vector The process is described by the following formula:

[0079]

[0080] (2) Input the optical flow map into the ConvLSTM network F ConvLSTM , output the corresponding time series feature vector And the state vector h of the networki-1 Update, described by the following formula:

[0081]

[0082]

[0083] Among them, h i Indicates the current state of the network.

[0084] (3) The temporal feature vector and the spatial feature vector are concatenated in the channel dimension and input into the encoder F based on the gated convolutional neural network GatedCNN , output the predicted target pose vector It is described by the following formula:

[0085]

[0086] At this point, the automatic motion planning is completed, and the target pose vector predicted by the motion planning module and the force data F between the endoscope and the nasal tissue obtained by the force sensing module at each sampling time t are combined. t , and the current state S of the seven-degree-of-freedom manipulator t Enter the control module together.

[0087] Step 5: Use the force sensing module through the endoscope to obtain force feedback data between the endoscope and the nasal tissue.

[0088] Step 6: The control module obtains the motion vector Q of each joint of the robotic arm based on the compliant control method T Based on the motion planning and force feedback data, the robot is controlled to execute the motion planning and perform real-time automatic control of the 7-DOF manipulator. This process can be described by the following formula:

[0089]

[0090] Among them, S t Indicates the current state of the seven-degree-of-freedom robotic arm, F t Represents force feedback data.

[0091] At this point, the robot's automatic control is completed. During the endoscopic examination process, the endoscopic image can be obtained in real time, and the next target can be obtained through the motion planning module. Combined with the robot's own state and force feedback data, the robot can be safely and stably controlled to automatically complete the next action target. This process is repeated until the entire nasal endoscopy examination process is automatically completed.

[0092] A fully automatic robot-assisted nasal endoscopy system includes a nasal endoscope, an image preprocessing module, an optical positioning module, a motion planning module, a force sensing module, a control module and a seven-degree-of-freedom robotic arm.

[0093] Nasal endoscope, used to obtain endoscopic images of the inside of the nasal cavity and input them into the image preprocessing module;

[0094] An image preprocessing module is connected to the nasal endoscope and is used to perform image enhancement and optical flow estimation on the endoscopic image; and is connected to the action planning module and is used to input the obtained enhanced endoscopic image and optical flow map into the action planning module;

[0095] The optical positioning module is connected to the nasal endoscope to obtain the spatial position of the nasal endoscope; it is connected to the seven-degree-of-freedom robotic arm to determine the initial target position;

[0096] The motion planning module is connected to the image preprocessing module and is used to obtain and use the preprocessed image data to predict the target posture transformation; it is connected to the control module and is used to transmit the target posture change data to determine the robot's motion plan;

[0097] The force sensing module is connected to the nasal endoscope to measure the force data between the endoscope and the nasal tissue; it is connected to the control module to transmit and process force feedback data for robot control.

[0098] The control module is connected to the motion planning module to obtain and process the planned target motion sequence; connected to the force sensing module to obtain and process force sensing data; and connected to the seven-degree-of-freedom robotic arm to control the robot to complete the posture transformation target safely and stably.

[0099] The seven-degree-of-freedom robotic arm is connected to the control module to acquire and execute actions, connected to the nasal endoscope to clamp the endoscope; and connected to the optical positioning module to determine the initial position of the robot.

[0100] Therefore, the present invention adopts the above-mentioned fully automatic robot-assisted nasal endoscopy inspection method and system, realizes automatic path planning based on endoscopic images, and then uses a robotic arm to clamp the nasal endoscope and uses a control system to safely and stably adjust the spatial position of the endoscope, thereby realizing high-efficiency and high-stability nasal endoscopy automatic inspection function.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A fully automatic robot-assisted nasal endoscopy system, characterized in that: include Nasal endoscope, used to obtain endoscopic images of the inside of the nasal cavity and input them into the image preprocessing module; An image preprocessing module is connected to the nasal endoscope and is used to perform image enhancement and optical flow estimation on the endoscopic image; and is connected to the action planning module and is used to input the obtained enhanced endoscopic image and optical flow map into the action planning module; An optical positioning module is connected to the nasal endoscope and is used to obtain the spatial position of the nasal endoscope; Connected to a seven-degree-of-freedom robotic arm to determine the initial target pose; The motion planning module is connected to the image preprocessing module and is used to obtain and use the preprocessed image data to predict the target posture transformation; Connected to the control module to transmit target posture change data and determine the robot's motion plan; A force sensing module is connected to the nasal endoscope and is used to measure the force data between the endoscope and the nasal tissue; Connecting with the control module to transmit and process force feedback data for robot control; The control module is connected to the motion planning module to obtain and process the planned target motion sequence; and is connected to the force sensing module to obtain and process the force sensing data; Connected to the seven-degree-of-freedom robotic arm, it is used to control the robot to complete posture transformation goals safely and stably; A seven-degree-of-freedom robotic arm, connected to the control module, used to acquire and execute actions, and connected to the nasal endoscope, used to clamp the endoscope; Connected to the optical positioning module to determine the initial position of the robot.

2. A fully automatic robot-assisted nasal endoscopy system according to claim 1, characterized in that: The image preprocessing module includes an image enhancement unit and an optical flow estimation unit; An image enhancement unit, connected to the nasal endoscope, for improving the resolution of images acquired by the nasal endoscope; The optical flow estimation unit is connected to the nasal endoscope and is used to obtain the optical flow map of the endoscope image at adjacent moments.

3. The fully automatic robot-assisted nasal endoscopy system according to claim 1, characterized in that: The optical positioning module includes The rigid frame is used to install all the modules of the system. The optical locator and the locator host connection are set on the rigid frame; Among them, the optical locator is used to transmit and receive infrared signals, and a reference frame is set on the optical locator. Reference frame, used to reflect infrared signals emitted by the optical positioning instrument; The locator host is connected to the optical locator and is used for spatial coordinate calculation and transmission.

4. The fully automatic robot-assisted nasal endoscopy system according to claim 1, characterized in that: The motion planning module extracts the feature representation of the time dimension and the space dimension based on the enhanced endoscopic image and the optical flow map, and performs motion planning based on the feature representation of the time dimension and the space dimension to obtain the target posture. The motion planning module includes A spatial feature extraction unit, connected to the image preprocessing module, is used to extract feature representations of the endoscopic image in the spatial dimension; A temporal feature extraction unit, connected to the image preprocessing module, is used to extract feature representations on the time dimension contained in the endoscopic image sequence; The pose encoding unit is connected to the spatial feature extraction unit and the temporal feature extraction unit to predict the pose transformation vector and fuse the spatial and temporal features to represent the planned target pose.

5. The fully automatic robot-assisted nasal endoscopy system according to claim 2, characterized in that: The image enhancement unit uses a diffusion model to obtain two consecutive frames of endoscopic images I i and I i-1 Perform image enhancement operation, and the enhanced image is represented as At the same time, the optical flow estimation unit uses the FlowNet network based on the current endoscopic image I i and the endoscopic image I at the previous moment i-1 Generate the corresponding optical flow map, expressed as 6. A fully automatic robot-assisted nasal endoscopy system according to claim 4, characterized in that: The spatial feature extraction unit uses the spatial dimension features of the current endoscopic image and optical flow map of the convolutional neural network based on the attention mechanism to input the endoscopic image into the front attention mechanism module F Attention ResNet50 encoder F ResNet50 , output the corresponding spatial feature vector The process is described by the following formula: Specifically: In the attention mechanism module, the augmented image of size (H, W, C) The input image of size (C, H, W) is obtained by dimension transformation and an input image of size (W,C,H) Then, adaptive average pooling is used to process F based on the original image and the transformed image. Avg , global pooling processing F Max And the convolutional layer F Conv Calculate the corresponding weight matrix {A I , A J , A K }; Use the weight matrix to multiply the corresponding original image element by element, and then average the weighted sum to get the output image It is described by the following formula: Among them, A I Represents the input image The weight of A J Represents the input image The weight of A K Represents the input image The weight of At the same time, the optical flow map Input ResNet50 encoder and output the corresponding spatial feature vector The process is described by the following formula:

7. The fully automatic robot-assisted nasal endoscopy system according to claim 4, characterized in that: The temporal feature extraction unit extracts the temporal dimension features of the optical flow graph, including: Input the optical flow map into the ConvLSTM network F ConvLSTM , output the corresponding time series feature vector And the state vector h of the network i-1 Update, described by the following formula: Among them, h i Indicates the current state of the network.

8. A fully automatic robot-assisted nasal endoscopy system according to claim 6 or 7, characterized in that: The temporal feature vector and the spatial feature vector are concatenated in the channel dimension and input into the encoder F based on the gated convolutional neural network. GatedCNN , output the predicted target pose vector It is described by the following formula:

9. The fully automatic robot-assisted nasal endoscopy system according to claim 1, characterized in that: The control module obtains the motion vector Q of each joint of the robotic arm based on the compliant control method t , to perform real-time automatic control of the seven-degree-of-freedom robotic arm, which can be described by the following formula: Among them, S t Indicates the current state of the seven-degree-of-freedom robotic arm, F t Represents force feedback data.

10. A fully automatic robot-assisted nasal endoscopy method, characterized in that: The following steps are involved: The control module is used to control the robot to clamp the nasal endoscope to the initial position; The nasal endoscope obtains endoscopic images of the nasal cavity and inputs them into the image preprocessing module; The endoscopic image is enhanced by using an image enhancement unit provided on the image preprocessing module, and a corresponding optical flow map is generated based on the current endoscopic image and the endoscopic image at the previous moment by using an optical flow estimation unit provided on the image preprocessing module, thereby completing the preprocessing of the endoscopic image data; The enhanced endoscopic image and optical flow map are input into the motion planning module, feature representations of the time dimension and the space dimension are extracted from the enhanced endoscopic image and the optical flow map, and motion planning is performed based on the feature representations of the time dimension and the space dimension; Using a force sensing module through the endoscope to obtain force feedback data between the endoscope and nasal tissue; After the action planning module completes the automatic action planning, the target pose vector predicted by the action planning module and the force feedback data F obtained by the force sensing module at each sampling time t are obtained between the endoscope and the nasal tissue. t , and the current state S of the seven-degree-of-freedom manipulator t The data are input together into the control module, which controls the robot to execute the motion plan based on the motion planning and force feedback data.

Images