Robot automatic path generation method, system and medium

Abstract

The present invention discloses a method, system, and medium for automatic robot path generation. The method comprises: collecting a sequence of nasal endoscope images to be input; performing feature analysis on the nasal endoscope images using a nasal endoscope image feature extraction model to obtain features of the nasal endoscope image sequence, wherein the nasal endoscope image feature extraction model is trained using a natural image dataset; inputting the extracted nasal endoscope image sequence features into a nasal endoscope pose estimation model to estimate the pose matrix of the nasal endoscope, wherein the nasal endoscope pose estimation model is trained using multiple sets of image pose matrices; and combining the nasal endoscope pose matrices in a time series to obtain the robot's motion trajectory. The present invention can achieve real-time and accurate automatic path generation for the robot based on the image sequence.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical robots, and more specifically, relates to a method, system and medium for automatically generating a path for a robot. Background Art

[0002] With the continuous advancement of medical technology, endoscopic surgery has become an important method in minimally invasive surgery. Due to its advantages such as minimal damage to the human body and rapid recovery, nasal endoscopic surgery is widely used in the diagnosis and treatment of nasal diseases. However, the complex structure of the nasal cavity and the limited operating space make doctors rely on endoscopic images during surgery, which poses certain operational difficulties and risks. Therefore, the use of computer vision and robotics technology to achieve automatic navigation and path planning for nasal endoscopes has become an important research direction for improving surgical precision and safety.

[0003] Traditional automatic path generation tasks based on endoscopic images are mainly based on methods such as extracting image features and manually crafting templates, but this can lead to feature matching errors and long execution times. With the development of deep learning, the introduction of deep learning into this field has shown better performance than traditional methods.

[0004] Jake Sganga proposed a deep learning architecture called OffsetNet for accurately positioning bronchoscopes in the lungs in real time. This architecture enables real-time positioning of endoscopes within the dynamic and tortuous anatomy of the lung airways based on images, generating an endoscopic path.

[0005] Based on gastrointestinal endoscopy, Ozyoruk et al. proposed an unsupervised monocular depth and endoscopic pose estimation path generation method called Endo-SfMLearner, which combines a residual network with a spatial attention module to instruct the network to focus on distinguishable and highly textured tissue regions.

[0006] To address the problems of tissue deformation and narrow field of view in colonoscopy, Anita Rau proposed SimCol, a synthetic dataset for endoscope pose estimation during colonoscopy. This method also proposed a new method for predicting endoscope pose and generating paths by explicitly learning a bimodal distribution. This method, which follows the structure of a feature extractor and pose decoder, classifies endoscope motion as insertion or exit, and introduces a novel supervised regression network to extract features from both frames.

[0007] Different from the spatial attention mechanism introduced by Ozyoruk of EndoSLAM, Liu et al. embedded a dual attention mechanism in the network, and the introduced channel attention module can capture long-range context information in the channel dimension.

[0008] We found that current methods are tailored to gastrointestinal endoscopy, and the network architecture does not consider the joint features of two image frames. Due to the narrow spatial space, lighting issues, and limited texture features of nasal endoscopy images, there is currently no path generation model specifically designed for nasal endoscopy images. Therefore, to address these shortcomings, we need to design a new deep learning model for nasal endoscopy pose estimation. Summary of the Invention

[0009] The present invention is provided to solve the above-mentioned problems existing in the prior art. Therefore, a method, system and medium for automatic robot path generation are needed, which can generate the robot's operation path based solely on image sequences and realize accurate and intelligent real-time detection of the robot path.

[0010] According to a first embodiment of the present invention, a method for automatically generating a robot path is provided, the method comprising:

[0011] Acquire a nasal endoscope image sequence, perform feature analysis on the nasal endoscope image sequence using a nasal endoscope image feature extraction model, and acquire features of a single-frame nasal endoscope image;

[0012] Inputting multiple nasal endoscope single-frame image features in series into a nasal endoscope joint feature extraction model for joint feature analysis to obtain joint features of two nasal endoscope image frames, and assigning different weights to the joint features of the two nasal endoscope image frames and the nasal endoscope single-frame image features through an attention mechanism to obtain features with different weights;

[0013] Inputting the features with different weights into a nasal endoscope pose estimation model to predict a relative pose transformation vector between two input image frames, wherein the relative pose transformation vector includes a translation vector and a rotation vector, and the nasal endoscope pose estimation model is trained using a pose matrix of a real nasal endoscope image dataset;

[0014] Determining a pose estimation matrix of the nasal endoscope for each frame according to the relative pose transformation vector and the homogeneous matrix of the initial pose;

[0015] Combined with the pose estimation matrix of the nasal endoscope in each frame, automatic path generation of the robot is achieved.

[0016] Furthermore, the nasal endoscope image feature extraction model includes a pre-trained ResNet-18 network, which takes the nasal endoscope image sequence as input and obtains the nasal endoscope single-frame image features by extracting features from two nasal endoscope images separated by k frames.

[0017] Furthermore, by extracting features from two nasal endoscope images separated by k frames, the features of a single-frame nasal endoscope image are obtained, specifically including:

[0018] Through the first convolution and maximum pooling layers, the low-level features of the image are initially extracted and the image size is reduced. Four groups of residual blocks are used to extract features from the shallow to the deepest levels to obtain a channel feature map. The channel feature map is globally averaged pooled to obtain the single-frame image features of the nasal endoscope.

[0019] Furthermore, the nasal endoscope joint feature extraction model includes a joint feature extractor and an attention mechanism network. The joint feature extractor takes two serially connected nasal endoscope single-frame image features as input and outputs a joint feature map. The attention mechanism network is used to assign weights to the joint feature map and the corresponding nasal endoscope single-frame image features to obtain features with different weights.

[0020] Furthermore, the attention mechanism network includes a multi-layer perceptron and a softmax layer, the multi-layer perceptron includes one or more fully connected layers, the multi-layer perceptron is used to calculate the weight vector of the joint feature map and the corresponding nasal endoscope single-frame image feature, the weight vector is normalized through the softmax layer, and the joint feature map and the corresponding nasal endoscope single-frame image feature are multiplied element-by-element with the corresponding normalized weight vector to obtain features with different weights.

[0021] Furthermore, the nasal endoscope pose estimation model includes a pose decoder, which includes a convolutional layer, a pooling layer and a fully connected layer connected in sequence. The decoder maps the features of different weights into relative transformation vectors between predicted nasal endoscope poses.

[0022] Furthermore, the pose estimation matrix of the nasal endoscope in each frame is combined to realize automatic path generation of the robot, specifically including:

[0023] The relative pose transformation vector is converted into a homogeneous transformation matrix to represent the pose of each frame image in three-dimensional space. The homogeneous transformation matrices of all frames are combined in chronological order to form a complete trajectory.

[0024] According to a second technical solution of the present invention, a robot automatic path generation system is provided, the system comprising:

[0025] An image acquisition module is configured to acquire a sequence of nasal endoscope images to be input;

[0026] an image feature extraction module, configured to extract single-frame image features from the nasal endoscope image sequence using a pre-trained ResNet-18 model;

[0027] a joint feature extraction module configured to concatenate the extracted single-frame image features and perform joint feature extraction through a multidimensional information integration module based on a ResNet-34 structure, and to assign weights to the single-frame image features and the joint feature map using an attention mechanism to obtain features with different weights;

[0028] A pose decoding module is configured to predict a relative pose transformation vector between two input image frames based on the features with different weights, wherein the relative pose transformation vector includes a translation vector and a rotation vector;

[0029] The trajectory generation module is configured to calculate the relative posture transformation vector, and combine it with the homogeneous matrix of the initial posture to obtain the nasal endoscope posture matrix of each frame, and perform time-series combination of the nasal endoscope posture matrix of each frame to generate the robot's motion trajectory.

[0030] Furthermore, the image feature extraction module includes a nasal endoscope image feature extraction model training unit, a nasal endoscope image feature extraction model construction unit and a nasal endoscope image feature extraction unit;

[0031] The nasal endoscope image feature extraction model construction unit is used to construct an initial nasal endoscope image feature extraction model based on a pre-trained ResNet-18 deep residual network, a ResNet-34 deep residual network and an attention mechanism network;

[0032] The nasal endoscope image feature extraction model training unit is used to train the initial nasal endoscope image feature extraction model using a real human nasal endoscope dataset to obtain a nasal endoscope image feature extraction model;

[0033] The nasal endoscope image feature extraction unit is used to perform feature analysis on the nasal endoscope image to be analyzed through the nasal endoscope image feature extraction model to obtain nasal endoscope image features.

[0034] Furthermore, the posture decoding module includes a posture decoding model construction unit, a posture decoding model training unit and a posture extraction unit;

[0035] The posture decoding model construction unit is used to construct an initial posture decoding model, and the initial posture decoding model includes a convolutional layer, a pooling layer and a fully connected layer;

[0036] The posture decoding model training unit is used to train the posture decoding model using multiple sets of nasal endoscope posture matrices to obtain a posture decoding model;

[0037] The posture extraction unit is used to extract the posture of the nasal endoscope image features to be decoded through the posture decoding model to obtain the nasal endoscope posture matrix.

[0038] According to a third technical solution of the present invention, a readable storage medium is provided, wherein the readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described above.

[0039] The present invention has at least the following beneficial effects:

[0040] The present invention can realize automatic path generation of the robot according to the image sequence in real time and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of a method for automatic path generation of a robot according to an embodiment of the present invention is shown.

[0042] Figure 2 A specific implementation flow chart of a robot automatic path generation method according to an embodiment of the present invention is shown.

[0043] Figure 3 A structural diagram of a nasal endoscope pose estimation model according to an embodiment of the present invention is shown.

[0044] Figure 4 A structural diagram of a robot automatic path generation system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are further described in detail below with reference to the accompanying drawings and specific embodiments, but are not intended to limit the present invention. For the various steps described herein, if there is no necessity for a contextual relationship between each other, the order in which they are described as examples herein should not be regarded as limiting, and those skilled in the art should know that they can be adjusted in order as long as the logic between them is not destroyed, resulting in the inability to implement the entire process.

[0046] Example 1:

[0047] Figure 1 FIG. 1 shows a flow chart of a method for automatically generating a robot path according to an embodiment of the present invention. Figure 1As shown, an embodiment of the present invention proposes a robot automatic path generation method, which first collects a sequence of nasal endoscope images to be analyzed, analyzes two frames of images through an image feature extraction model to obtain features of the two frames of images; concatenates the features of the two separate frames of images through a joint feature extraction model for joint feature extraction, and then assigns different weights to the separate features and the joint features; passes the features of different weights through a posture decoding model to obtain a relative posture transformation vector between the two input image frames; and combined with the initial posture matrix, the trajectory generation model performs time-series joint on the relative posture matrix of the nasal endoscope to generate the robot's motion trajectory.

[0048] Specifically, Figure 2 FIG. 4 shows a specific implementation flow chart of a method for automatically generating a robot path according to an embodiment of the present invention. Figure 2 As shown, the robot automatic path generation method includes the following steps:

[0049] Step S100 , obtaining a nasal endoscope image sequence, performing feature analysis on the nasal endoscope image sequence using a nasal endoscope image feature extraction model, and obtaining features of a single-frame nasal endoscope image.

[0050] In this embodiment, the nasal endoscope image feature extraction model mainly uses the pre-trained ResNet-18. ResNet-18 is a variant of the Residual Network and contains 18 layers. This network solves the gradient vanishing problem in deep network training by introducing residual connections, so that image features can be effectively extracted. The nasal endoscope image is adjusted to a 224×224 RGB format and input into the ResNet-18 model. Through the first layer of convolution and the maximum pooling layer, the low-level features of the image are preliminarily extracted and the image size is reduced. Then, four groups of residual blocks are used to extract features from the shallow to the deepest level. The channel feature map output by the residual block is globally average pooled to obtain a feature vector, which is input as the feature of a single frame image into the subsequent module or subsequent steps for processing.

[0051] Step S200, multiple nasal endoscope single-frame image features are input in series into a nasal endoscope joint feature extraction model for joint feature analysis to obtain joint features of two nasal endoscope images, and different weights are assigned to the joint features of the two nasal endoscope images and the nasal endoscope single-frame image features through an attention mechanism to obtain features with different weights.

[0052] In this embodiment, the nasal endoscope joint feature extraction model is a network structure based on ResNet-34, which is used to integrate multiple single-frame image features and extract more representative joint features. ResNet-34 is similar to ResNet-18, but has a deeper network layer to obtain stronger feature extraction capabilities. ResNet-34 consists of a 34-layer network, including a convolutional layer, a batch normalization layer, a ReLU activation layer, a maximum pooling layer, and a residual block. The two single-frame image features extracted by ResNet-18 are concatenated to form a new feature vector. The concatenated feature vector is input into the joint feature model based on ResNet-34, and deep features are further extracted through multiple convolutional layers.

[0053] Features are processed using multiple layers of residual blocks to extract more representative joint features. The output feature map is then subjected to global average pooling to generate a joint feature vector. This joint feature vector is used in the subsequent attention mechanism. In a neural network, the importance of different features and neurons varies. The final output is likely determined by only a few individual features, and irrelevant features may negatively impact the final result. The attention mechanism selects features by assigning weights, with more important features being assigned greater weights. The attention mechanism consists of a multi-layer perceptron (MLP) and a softmax layer. The joint feature vector and the feature vector of a single image are input. The MLP consists of one or more fully connected layers, which calculate feature weight vectors. The weight vectors are normalized by the softmax layer to obtain the weight value for each feature. The joint feature vector and the feature vector of a single image are element-wise multiplied by the normalized weight vector to obtain the weighted feature vector.

[0054] Step S300, input the features with different weights into the nasal endoscope pose estimation model, predict the relative pose transformation vector between the two input image frames, the relative pose transformation vector includes a translation vector and a rotation vector, and the nasal endoscope pose estimation model is obtained by training the pose matrix of the real nasal endoscope image dataset.

[0055] Step S400: determining a pose estimation matrix of the nasal endoscope for each frame according to the relative pose transformation vector and the homogeneous matrix of the initial pose.

[0056] Step S500 , combining the pose estimation matrix of the nasal endoscope in each frame to realize automatic path generation of the robot.

[0057] In this embodiment, the structure of the nasal endoscope posture estimation model is as follows: Figure 3As shown in the figure, it is used to convert the extracted features into a pose estimation matrix for the nasal endoscope. The pose decoding model consists of multiple convolutional layers, activation functions, normalization layers, and fully connected layers. Its detailed structure is shown in the figure. The weighted feature vector obtained from the joint feature model and the attention mechanism is decoded into a 1x6 pose transformation vector, including a translation vector and a rotation vector.

[0058] The pose matrix vector describes the position and rotation of each frame in 3D space. The relative pose transformation vector includes a translation vector and a rotation vector, for a total of 6 degrees of freedom. The trajectory generation model converts the relative pose transformation vector into a homogeneous transformation matrix, representing the pose of each frame in 3D space. The homogeneous transformation matrices of all frames are combined in chronological order to form a complete trajectory.

[0059] All modules are trained end-to-end using a real-world human nasal endoscope dataset. The dataset includes observed nasal endoscope image sequences, endoscope trajectories, and other information.

[0060] Example 2:

[0061] Figure 4 The structure diagram of a robot automatic path generation system according to an embodiment of the present invention is shown. The embodiment of the present invention provides a robot automatic path generation system, such as Figure 4 As shown, the system 400 includes an image acquisition module 401 , an image feature extraction module 402 , a joint feature extraction module 403 , a pose decoding module 404 and a trajectory generation module 405 .

[0062] The image acquisition module 401 uses a XION binocular 4K endoscope to acquire nasal endoscope images; the image feature extraction module is used to construct a feature extraction model for a single-frame image, and perform image feature analysis on the input single-frame image of the nasal endoscope through the image feature extraction model to obtain the image features of each of the two frames of image respectively; the joint feature extraction module is used to construct a joint feature extraction model, and obtain the joint feature information by connecting the extracted features of the two frames of image in series, and then assigning different weights to the individual features and the joint features; the posture decoding module is used to decode the features of the previous different weights to predict the relative posture transformation vector between the two input image frames; the trajectory generation module combines the initial posture matrix, performs time-series joint on the relative posture matrix of the nasal endoscope, and generates the motion trajectory of the robot.

[0063] The image feature extraction module 402 includes a nasal endoscope image feature extraction model training unit, a nasal endoscope image feature extraction model construction unit and a nasal endoscope image feature extraction unit; the nasal endoscope image feature extraction model construction unit is used to construct an initial nasal endoscope image feature extraction model based on a pre-trained ResNet-18 deep residual network; the nasal endoscope image feature extraction model training unit is used to train the initial nasal endoscope image feature extraction model through a real human nasal endoscope dataset to obtain a nasal endoscope image feature extraction model; the nasal endoscope image feature extraction unit is used to perform feature analysis on the nasal endoscope image to be analyzed through the nasal endoscope image feature extraction model to obtain single-frame nasal endoscope image features.

[0064] The joint feature extraction module 403 includes a joint feature extraction model training unit for nasal endoscope images, a joint feature extraction model construction unit for nasal endoscope images, and a joint feature extraction unit for nasal endoscope images. The joint feature extraction model construction unit is used to construct an initial joint feature extraction model for nasal endoscope images based on a pre-trained ResNet-34 deep residual network and an attention mechanism network. The joint feature extraction model training unit is used to train the initial joint feature extraction model for nasal endoscope images using a real human nasal endoscope dataset to obtain a joint feature extraction model for nasal endoscope images. The joint feature extraction unit is used to concatenate image features for joint feature extraction to obtain a joint feature map. The attention mechanism is used to assign weights to single-frame image features and the joint feature map to obtain features with different weights.

[0065] The posture decoding module 404 includes a posture decoding model construction unit, a posture decoding model training unit and a posture extraction unit; the posture decoding model construction unit is composed of a convolutional layer, a pooling layer and a fully connected layer, etc., to construct an initial posture decoding model; the posture decoding model training unit is used to train the posture decoding model through multiple sets of nasal endoscope posture matrices to obtain the posture decoding model; the posture extraction unit is used to perform posture extraction on the nasal endoscope image features to be decoded through the posture decoding model to obtain the nasal endoscope posture matrix.

[0066] An embodiment of the present invention further provides a readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the methods described in the above embodiments.

[0067] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of their solutions) can be used in combination with each other. For example, those of ordinary skill in the art may use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features can be grouped together to simplify the present invention. This should not be interpreted as an intention that a feature of an invention that is not claimed for protection is necessary for any claim. On the contrary, the subject matter of the present invention may be less than all the features of the embodiments of a particular invention. Thus, the following claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim is independently a separate embodiment, and it is considered that these embodiments can be combined with each other in various combinations or arrangements. The scope of the present invention should be determined with reference to the appended claims and the full scope of equivalents to which these claims are entitled.

Claims

1. A method for automatically generating a robot path, characterized in that: The method comprises: Acquire a nasal endoscope image sequence, perform feature analysis on the nasal endoscope image sequence using a nasal endoscope image feature extraction model, and acquire features of a single-frame nasal endoscope image; Inputting multiple nasal endoscope single-frame image features in series into a nasal endoscope joint feature extraction model for joint feature analysis to obtain joint features of two nasal endoscope image frames, and assigning different weights to the joint features of the two nasal endoscope image frames and the nasal endoscope single-frame image features through an attention mechanism to obtain features with different weights; Inputting the features with different weights into a nasal endoscope pose estimation model to predict a relative pose transformation vector between two input image frames, wherein the relative pose transformation vector includes a translation vector and a rotation vector, and the nasal endoscope pose estimation model is trained using a pose matrix of a real nasal endoscope image dataset; Determining a pose estimation matrix of the nasal endoscope for each frame according to the relative pose transformation vector and the homogeneous matrix of the initial pose; Combined with the pose estimation matrix of the nasal endoscope in each frame, automatic path generation of the robot is achieved.

2. The method according to claim 1, characterized in that The nasal endoscope image feature extraction model includes a pre-trained ResNet-18 network. The ResNet-18 network takes the nasal endoscope image sequence as input and obtains the features of a single-frame nasal endoscope image by extracting features from two nasal endoscope images separated by k frames.

3. The method according to claim 2, characterized in that By extracting features from two nasal endoscope images separated by k frames, the features of a single nasal endoscope image are obtained, specifically including: Through the first convolution and maximum pooling layers, the low-level features of the image are initially extracted and the image size is reduced. Four groups of residual blocks are used to extract features from the shallow to the deepest levels to obtain a channel feature map. The channel feature map is globally averaged pooled to obtain the single-frame image features of the nasal endoscope.

4. The method according to claim 1, wherein The nasal endoscope joint feature extraction model includes a joint feature extractor and an attention mechanism network. The joint feature extractor takes two serially connected nasal endoscope single-frame image features as input and outputs a joint feature map. The attention mechanism network is used to assign weights to the joint feature map and the corresponding nasal endoscope single-frame image features to obtain features with different weights.

5. The method according to claim 4, characterized in that The attention mechanism network includes a multi-layer perceptron and a softmax layer, and the multi-layer perceptron includes one or more fully connected layers. The multi-layer perceptron is used to calculate the weight vector of the joint feature map and the corresponding nasal endoscope single-frame image feature, and the weight vector is normalized by the softmax layer. The joint feature map and the corresponding nasal endoscope single-frame image feature are multiplied element-by-element with the corresponding normalized weight vector to obtain features with different weights.

6. The method according to claim 1, characterized in that The nasal endoscope pose estimation model includes a pose decoder, which includes a convolutional layer, a pooling layer and a fully connected layer connected in sequence. The decoder maps the features of different weights into relative transformation vectors between predicted nasal endoscope poses.

7. The method according to claim 6, characterized in that The pose estimation matrix of the nasal endoscope in each frame is combined to realize the automatic path generation of the robot, specifically including: The relative pose transformation vector is converted into a homogeneous transformation matrix to represent the pose of each frame image in three-dimensional space. The homogeneous transformation matrices of all frames are combined in chronological order to form a complete trajectory.

8. A robot automatic path generation system, characterized in that: The system comprises: An image acquisition module is configured to acquire a sequence of nasal endoscope images to be input; an image feature extraction module, configured to extract single-frame image features from the nasal endoscope image sequence using a pre-trained ResNet-18 model; a joint feature extraction module configured to concatenate the extracted single-frame image features and perform joint feature extraction through a multi-dimensional information integration module based on a ResNet-34 structure, and to assign weights to the single-frame image features and the joint feature map using an attention mechanism to obtain features with different weights; A pose decoding module is configured to predict a relative pose transformation vector between two input image frames based on the features with different weights, wherein the relative pose transformation vector includes a translation vector and a rotation vector; The trajectory generation module is configured to calculate the relative posture transformation vector, and combine it with the homogeneous matrix of the initial posture to obtain the nasal endoscope posture matrix of each frame, and perform time-series combination of the nasal endoscope posture matrix of each frame to generate the robot's motion trajectory.

9. The system according to claim 8, characterized in that The image feature extraction module includes a nasal endoscope image feature extraction model training unit, a nasal endoscope image feature extraction model construction unit and a nasal endoscope image feature extraction unit; The nasal endoscope image feature extraction model construction unit is used to construct an initial nasal endoscope image feature extraction model based on a pre-trained ResNet-18 deep residual network, a ResNet-34 deep residual network and an attention mechanism network; The nasal endoscope image feature extraction model training unit is used to train the initial nasal endoscope image feature extraction model using a real human nasal endoscope dataset to obtain a nasal endoscope image feature extraction model; The nasal endoscope image feature extraction unit is used to perform feature analysis on the nasal endoscope image to be analyzed through the nasal endoscope image feature extraction model to obtain nasal endoscope image features.

10. A readable storage medium, characterized in that: The readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 7.

Images