Digestive tract catheterization robot navigation method and system based on multi-modal information fusion

Abstract

The invention relates to the technical field of medical robots, in particular to an alimentary canal catheterization robot navigation method and system based on multi-modal information fusion. The method comprises the following steps: carrying out real-time data acquisition on a human body cavity by utilizing a multi-mode sensor; the collected multi-modal data is preprocessed; synthesizing multi-modal data, extracting cavity environment characteristics, and establishing a three-dimensional model of a human body cavity in real time; human body cavity environment characteristics are analyzed, and local and global paths are generated; outputting the path to a catheterization robot controller, driving a catheterization robot, monitoring a navigation abnormal state in real time, and collecting and storing the path, the sensor state and operation data in the navigation process; the system comprises a data acquisition module, a data processing module, a three-dimensional model establishment module, a path generation module and a data monitoring module. Through the mode, the environment is comprehensively perceived, the navigation precision and the real-time performance are improved, and the high-precision operation requirement in a clinical complex environment is met.

Description

Technical Field

[0001] The present invention relates to the technical field of medical robots, and in particular to a digestive tract catheterization robot navigation method and system based on multimodal information fusion. Background Art

[0002] At present, gastrointestinal catheterization technology is widely used in gastroscopy, colonoscopy and other interventional operations, which can provide key support for clinical diagnosis and treatment. However, traditional catheterization methods mainly rely on the experience of medical staff and a single type of sensor, such as vision or force perception, which is easily affected by factors such as the complexity of the cavity environment, the variability of organ morphology and the physiological differences of patients during the operation.

[0003] Although the existing catheterization robot system has introduced certain automation technologies, it usually relies on single-modal sensors to obtain environmental information, such as simply using an endoscopic camera to capture the cavity morphology or only using a force sensor to monitor the contact force. This type of method has limitations in data dimensions and it is difficult to fully reflect the detailed characteristics of the cavity environment. In addition, single-modal sensors have large noise and errors, which can easily lead to inaccurate navigation or misjudgment of high-risk areas. Existing technologies have obvious deficiencies in the following aspects:

[0004] 1. Limited environmental perception capabilities: A single sensor is difficult to comprehensively reflect the geometry, physical properties and dynamic changes of the cavity, resulting in incomplete navigation data.

[0005] 2. Path planning lacks intelligence: Most existing systems only support static or local path planning and fail to combine multimodal information for real-time optimization, which makes it easy for navigation failures or deviations to occur in complex cavity environments.

[0006] 3. Lack of real-time performance: Due to the low efficiency of information fusion and processing, existing methods are difficult to meet real-time requirements, especially in complex cavity environments.

[0007] In summary, the existing gastrointestinal catheterization technology has problems such as incomplete environmental perception, low navigation accuracy, and poor real-time performance, which makes it difficult to meet the high-precision operation requirements in complex clinical environments. Summary of the invention

[0008] The purpose of the present invention is to provide a gastrointestinal catheterization robot navigation method and system based on multimodal information fusion, aiming to solve the technical problems of the existing gastrointestinal catheterization technology in the prior art, such as incomplete environmental perception, low navigation accuracy, and poor real-time performance, and it is difficult to meet the high-precision operation requirements in complex clinical environments.

[0009] To achieve the above object, the present invention adopts a digestive tract catheterization robot navigation method based on multimodal information fusion, which comprises the following steps:

[0010] Using multimodal sensors to collect real-time data of human cavities; the multimodal sensors include visual sensors, force sensors, ultrasonic sensors and inertial measurement units;

[0011] Preprocess the collected multimodal data;

[0012] Integrate multimodal data, extract cavity environment characteristics, and build a three-dimensional model of the human cavity in real time;

[0013] Analyze the characteristics of the human cavity environment and generate local and global paths;

[0014] The path is output to the intubation robot controller to drive the intubation robot, monitor the navigation abnormality in real time, and collect and store the path, sensor status and operation data during the navigation process.

[0015] Among them, in the step of using a multimodal sensor to collect real-time data of a human cavity:

[0016] The visual sensor is used to capture image information inside the cavity and identify the cavity shape, texture and dynamic changes;

[0017] The force sensor is used to monitor the contact force and direction between the catheter and the cavity wall to determine whether there is abnormal pressure;

[0018] The ultrasonic sensor is used to obtain the depth information and physical characteristics of the cavity tissue;

[0019] The inertial measurement unit is used to record the posture and motion trajectory of the catheter in the cavity and monitor the acceleration and angular velocity.

[0020] Among them, in the step of preprocessing the collected multimodal data:

[0021] Preprocessing methods include noise filtering, time synchronization and coordinate alignment.

[0022] Among them, in the step of integrating multimodal data, extracting cavity environment features, and establishing a three-dimensional model of the human cavity in real time:

[0023] The time features and motion dynamic features are extracted respectively, and the fused features are mapped into the three-dimensional geometric structure of the cavity through a three-dimensional modeling algorithm based on voxel representation, the cavity surface mesh is generated, and the three-dimensional model is output.

[0024] Among them, in the step of analyzing the characteristics of the human body cavity environment and generating local and global paths, the process of generating the local path is:

[0025] For each neighbor node n′ of the current node n, calculate its temporary cost g′(n′)=g(n)+c(n,n′); where c(n,n′) is the cost of moving between nodes;

[0026] Define inaccessible areas and dynamically adjust the route to avoid them;

[0027] Determine changes in the environment ahead and make real-time adjustments.

[0028] Among them, in the step of calculating the temporary cost g′(n′)=g(n)+c(n,n′) for each neighbor node n′ of the current node n:

[0029] If a neighbor node n′ is not in the open list and a better path is found, then f′(n′)=g′(n′)+h(n′), update g(n′) and f(n′), and set the parent node of n′ to n.

[0030] Among them, in the step of calculating the temporary cost g′(n′)=g(n)+c(n,n′) for each neighbor node n′ of the current node n:

[0031] If the neighbor node n' is not in the open list, add it to the open list.

[0032] Among them, in the step of analyzing the characteristics of the human body cavity environment and generating local and global paths, the process of generating the global path is:

[0033] The human cavity is represented as a weighted undirected graph G(V,E); where V represents the cavity node set, E represents the edge set connecting the nodes, and the edge weight ω(e) represents the travel cost between two nodes;

[0034] Based on the overall human body cavity, the location of the target area, the size of the navigation device and the overall passability of the cavity are determined to generate a global path.

[0035] The present invention also provides a digestive tract catheterization robot navigation system based on multimodal information fusion, comprising a data acquisition module, a data processing module, a three-dimensional model building module, a path generation module and a data monitoring module; wherein:

[0036] The data acquisition module is used to collect real-time data of human cavities using a multimodal sensor;

[0037] The data processing module is used to preprocess the collected multimodal data; wherein the preprocessing method includes noise filtering, time synchronization and coordinate alignment;

[0038] The three-dimensional model building module is used to integrate multi-modal data, extract cavity environment characteristics, and build a three-dimensional model of the human cavity in real time;

[0039] The path generation module is used to analyze the characteristics of the human body cavity environment and generate local and global paths;

[0040] The data monitoring module is used to output the path to the tube placement robot controller, drive the tube placement robot, monitor the navigation abnormality status in real time, and collect and store the path, sensor status and operation data during the navigation process.

[0041] The present invention discloses a digestive tract catheterization robot navigation method and system based on multimodal information fusion, which collects real-time data of the human cavity by utilizing multimodal sensors; wherein the multimodal sensors include visual sensors, force sensors, ultrasonic sensors and inertial measurement units; preprocesses the collected multimodal data; integrates the multimodal data, extracts cavity environment characteristics, and establishes a three-dimensional model of the human cavity in real time; analyzes the human cavity environment characteristics, and generates local and global paths; outputs the path to the catheterization robot controller, drives the catheterization robot, monitors the navigation abnormality in real time, collects and stores the path, sensor status and operation data during the navigation process; realizes comprehensive perception of the environment, improves navigation accuracy and real-time performance, and meets the high-precision operation requirements in complex clinical environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flowchart of the steps of the digestive tract catheterization robot navigation method based on multimodal information fusion of the present invention.

[0044] Figure 2 It is a flow chart of the steps of generating a local path of the present invention.

[0045] Figure 3 It is a flow chart of the steps of generating a global path of the present invention.

[0046] Figure 4 It is a structural principle diagram of the digestive tract catheterization robot navigation system based on multimodal information fusion of the present invention.

[0047] Figure 5 It is a structural principle diagram of the electronic equipment of the present invention.

[0048] 601-data acquisition module, 602-data processing module, 603-three-dimensional model building module, 604-path generation module, 605-data monitoring module. DETAILED DESCRIPTION

[0049] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0050] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0051] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] See also Figure 1 to Figure 3 ,in Figure 1 This is a flowchart of the steps of the digestive tract catheterization robot navigation method based on multimodal information fusion. Figure 2 is a flowchart of the steps to generate a local path, Figure 3 is a flow chart of the steps to generate a global path.

[0053] The present invention provides a digestive tract catheterization robot navigation method based on multimodal information fusion, comprising the following steps:

[0054] S100: Use multimodal sensors to collect real-time data on human cavities; the multimodal sensors include visual sensors, force sensors, ultrasonic sensors and inertial measurement units.

[0055] In this embodiment, the visual sensor is used to capture the image information inside the cavity and identify the cavity morphology, texture and dynamic changes; for example, an endoscope camera is used to continuously shoot the video stream inside the cavity, and the image data is transmitted in real time. The image acquisition frequency is dynamically adjusted according to the complexity of the cavity environment to ensure that key features are not missed when moving at high speed; the image data is transmitted in the form of a video stream, and the frame rate is determined by the complexity of the cavity environment C e Dynamic adjustment, the relationship between the frame rate F can be expressed as:

[0056] F=max(F min ,k·C e ).

[0057] The force sensor is used to monitor the contact force and direction between the catheter and the cavity wall to determine whether there is abnormal pressure; by detecting the contact force between the front end of the catheter and the cavity wall, and transmitting the force / torque data of six dimensions to the processing module F:

[0058] F=[F x ,F y ,F z ,T x ,T y ,T z ] T ;

[0059] By sampling at a high frequency, such as 1kHz, subtle force changes can be captured, facilitating subsequent analysis of potential risk areas.

[0060] The ultrasonic sensor is used to obtain the depth information and physical characteristics of the cavity tissue; for example, an ultrasonic probe is used to transmit an ultrasonic signal, and after receiving the echo, the depth distribution and tissue characteristics of the cavity are calculated to generate the tomographic information. The acquisition frequency can be adjusted according to the detection accuracy requirements, for example, the sampling density can be increased in narrow or complex cavities to calculate the depth distribution d(x,y) and physical characteristics ρ(x,y) of the cavity tissue. The depth d(x,y) is calculated by the ultrasonic propagation time v and the sound speed t:

[0061]

[0062] The inertial measurement unit is used to record the posture and motion trajectory of the catheter in the cavity, monitor the acceleration and angular velocity, and record the three-dimensional acceleration of the catheter a = [a x ,a y ,a z ] and angular velocity ω=[ω x ,ω y ,ω z ]. Combined with the timestamp t, the motion trajectory and posture of the catheter can be updated in real time through the following integral formula:

[0063]

[0064] Where v(t) and r(t) are velocity and position respectively.

[0065] The above real-time integration and storage of multimodal sensors:

[0066] M(t k )={I(t k ),F c (t k ),T c (t k ),d(x,y,t k ),a(t k ),ω(t k )};

[0067] Where {I(t k ) is the image data, F c (t k ),T c (t k ) is the force data, d(x,y,t k ) is the ultrasonic depth information, a(t k ),ω(t k ) is the motion information.

[0068] S200: Preprocessing the collected multimodal data; wherein the preprocessing method includes noise filtering, time synchronization and coordinate alignment.

[0069] In this embodiment, noise filtering, time synchronization and coordinate alignment are used to improve data quality and ensure the spatiotemporal consistency of multimodal data. The image data is smoothed using a Gaussian filter, which is:

[0070]

[0071] A low-pass filter is applied to remove the interference introduced by high-frequency vibration in the force data. The low-pass filter is:

[0072]

[0073] A synchronization method based on hardware timestamp is used to align the data of different sensors to a unified time base, and Kalman filtering is applied to the timestamps for filtering.

[0074] Through sensor calibration and rigid transformation, the data of each modality are aligned to a unified reference coordinate system. After the above preprocessing steps, the original multimodal data are converted into a high-quality, temporally and spatially consistent format, which is convenient for subsequent feature extraction and modeling analysis.

[0075] S300: Integrate multimodal data, extract cavity environment characteristics, and establish a three-dimensional model of the human cavity in real time.

[0076] In this embodiment, time features and motion dynamic features are extracted respectively, and the fused features are mapped to the three-dimensional geometric structure of the cavity through a three-dimensional modeling algorithm based on voxel representation, the cavity surface mesh is generated, and the three-dimensional model is output.

[0077] Among them, a convolutional neural network is used to extract the spatial features in the cavity image. The forward propagation process of the convolutional neural network is as follows:

[0078] y (l) =f(W (l) *x (l-1) +b (l) );

[0079] Among them, y (l) is the output feature of the lth layer, W (l) is the convolution kernel, b (l) is the bias, f is the activation function (such as ReLU), * represents the convolution operation, x (l-1) is the input of the previous layer.

[0080] Force perception and ultrasound time series data processing, using bidirectional long short-term memory network (Bi-LSTM) to extract time-related features.

[0081] Inertial data feature extraction uses a spatiotemporal convolutional network to process acceleration and angular velocity data to capture dynamic motion features.

[0082] The above extracted features are fused through a multi-layer fully connected network to form a unified cavity feature representation. The operation of the fusion layer can be expressed as:

[0083] F fusion =f(W fusion ·[F vsion ,F force ,F ultrasound ,F interial ]+b fusion );

[0084] Among them, F fusion is the fused feature representation, F vsion ,F force ,F ultrasound ,F interial are visual, force, ultrasonic and inertial features respectively, W fusion and b fusion are weights and biases, and f is the activation function.

[0085] The fused features are mapped to the 3D geometric structure of the cavity through a 3D modeling algorithm based on voxel representation. The intensity value of each voxel V(i,j,k) is calculated from the fused features:

[0086] V(i,j,k)=φ(F fusion ).

[0087] The MarchingCube algorithm is used to generate the cavity surface mesh. The algorithm steps are as follows:

[0088] Vertex Position=Interp(V low ,V high ,Threshold);

[0089] Combining the above-mentioned deep learning algorithm and three-dimensional modeling method, the real-time generated cavity model can accurately reflect the spatial morphology and dynamic changes of the cavity, providing an important basis for subsequent path planning and navigation.

[0090] S400: Analyze the characteristics of the human cavity environment and generate local and global paths;

[0091] Furthermore, the process of generating a local path is:

[0092] 411: For each neighbor node n′ of the current node n, calculate its temporary cost g′(n′)=g(n)+c(n,n′); where c(n,n′) is the cost of moving between nodes;

[0093] 412: Define inaccessible areas and dynamically adjust the path to avoid inaccessible areas;

[0094] 413: Determine if the environment ahead has changed and make real-time adjustments.

[0095] Furthermore, if the neighbor node n′ is not in the open list and a better path is found, f′(n′)=g′(n′)+h(n′), g(n′) and f(n′) are updated, and the parent node of n′ is set to n.

[0096] Furthermore, if the neighbor node n′ is not in the open list, it is added to the open list.

[0097] Furthermore, the process of generating the global path is:

[0098] 421: The human cavity is represented as a weighted undirected graph G(V,E); where V represents the cavity node set, E represents the edge set connecting the nodes, and the edge weight ω(e) represents the travel cost between two nodes;

[0099] 422: Based on the overall human body cavity, determine the location of the target area, the size of the navigation device, and the overall passability of the cavity to generate a global path.

[0100] In this embodiment, based on the cavity model established in real time, the local path planning module adopts the A* algorithm based on heuristic search to dynamically generate a local optimal path according to the current position of the navigation device and the cavity environment characteristics perceived in real time.

[0101] The goal of local path planning is to minimize the total cost function from the starting point to the target point, f(n) = g(n) + h(n), where g(n) is the cost from the starting point to the current node n, and h(n) is the estimated cost from the current node n to the target node.

[0102] For each neighbor node n' of the current node n, calculate its temporary cost g'(n') = g(n) + c(n,n'), where c(n,n') is the cost of moving between nodes. If the neighbor node is not in the open list, or a better path is found, f'(n') = g'(n') + h(n'), update g(n') and f(n'), and set n''s parent node to n. If n' is not in the open list, add it.

[0103] According to the cavity model data collected by the multimodal sensor, the inaccessible areas are defined, and the path is dynamically adjusted to avoid these areas. For the heuristic estimate h(n), the Euclidean distance or the weighted Manhattan distance is usually used for calculation.

[0104] If the environment in front of the navigation device changes, such as new obstacles appearing or the path being blocked, the D* algorithm is used to make real-time adjustments to the existing path to ensure that the device always moves along the optimal path.

[0105] The global path planning module is based on the overall cavity anatomical model, comprehensively considering the location of the target area, the size of the navigation device and the overall passability of the cavity, and uses the Dijkstra algorithm or algorithm based on graph search to generate a global path.

[0106] The human cavity is represented as a weighted undirected graph G(V,E), where V represents the cavity node set, E represents the edge set connecting the nodes, and the edge weight ω(e) represents the travel cost between two nodes.

[0107] In the actual navigation process, the navigation device may encounter dynamic changes in the cavity or abnormal sensor data. Therefore, when the device detects that the path is blocked or the target position changes, a new local path is generated through the local path planning module and the global path is updated to ensure the navigation process of the device is safe and efficient.

[0108] S500: Output the path to the intubation robot controller, drive the intubation robot, monitor the navigation abnormality status in real time, collect and store the path, sensor status and operation data during the navigation process.

[0109] In this embodiment, feedback control and motion control strategies are used to drive the catheter placement robot to achieve precise navigation and complete the catheter placement operation; an abnormality detection mechanism is established to monitor high-risk signals during the navigation process in real time, and initiate emergency processing when necessary; the path, sensor status and operation data during the navigation process are collected, output, displayed and recorded for subsequent analysis and system optimization.

[0110] Corresponding to the aforementioned embodiment of the digestive tract catheterization robot navigation method based on multimodal information fusion, the present application also provides an embodiment of the digestive tract catheterization robot navigation system based on multimodal information fusion.

[0111] Figure 4 is a block diagram of a digestive tract catheterization robot navigation system based on multimodal information fusion according to an exemplary embodiment. Figure 4 The system may include: a data acquisition module 601, a data processing module 602, a three-dimensional model building module 603, a path generation module 604 and a data monitoring module 605; wherein:

[0112] The data acquisition module 601 is used to collect real-time data of human body cavities using a multimodal sensor;

[0113] The data processing module 602 is used to pre-process the collected multi-modal data; wherein the pre-processing method includes noise filtering, time synchronization and coordinate alignment;

[0114] The three-dimensional model building module 603 is used to integrate multi-modal data, extract cavity environment characteristics, and build a three-dimensional model of the human cavity in real time;

[0115] The path generation module 604 is used to analyze the characteristics of the human body cavity environment and generate local and global paths;

[0116] The data monitoring module 605 is used to output the path to the intubation robot controller, drive the intubation robot, monitor the navigation abnormality status in real time, and collect and store the path, sensor status and operation data during the navigation process.

[0117] In this embodiment, the data acquisition module 601 uses a multimodal sensor to perform real-time data acquisition on the human cavity; the data processing module 602 preprocesses the collected multimodal data; the preprocessing method includes noise filtering, time synchronization and coordinate alignment; the three-dimensional model establishment module 603 integrates multimodal data, extracts cavity environment characteristics, and establishes a three-dimensional model of the human cavity in real time; the path generation module 604 analyzes the human cavity environment characteristics and generates local and global paths; the data monitoring module 605 outputs the path to the catheter placement robot controller, drives the catheter placement robot, and monitors the navigation abnormality in real time, collects and stores the path, sensor status and operation data during the navigation process; it realizes comprehensive perception of the environment, improves navigation accuracy and real-time performance, and meets the high-precision operation requirements in complex clinical environments.

[0118] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0119] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.

[0120] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned digestive tract catheterization robot navigation method based on multimodal information fusion. Figure 5 As shown in FIG. 1 , a hardware structure diagram of a digestive tract catheterization robot navigation system based on multimodal information fusion provided by an embodiment of the present invention is provided for any device with data processing capability, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0121] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the digestive tract catheterization robot navigation method based on multimodal information fusion as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0122] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0123] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A digestive tract catheterization robot navigation method based on multimodal information fusion, characterized in that: The steps include: Using multimodal sensors to collect real-time data of human cavities; the multimodal sensors include visual sensors, force sensors, ultrasonic sensors and inertial measurement units; Preprocess the collected multimodal data; Integrate multimodal data, extract cavity environment characteristics, and build a three-dimensional model of the human cavity in real time; Analyze the characteristics of the human cavity environment and generate local and global paths; The path is output to the intubation robot controller to drive the intubation robot, monitor the navigation abnormality in real time, and collect and store the path, sensor status and operation data during the navigation process.

2. The digestive tract catheterization robot navigation method based on multimodal information fusion according to claim 1, characterized in that: In the step of using a multimodal sensor to collect real-time data of a human cavity: The visual sensor is used to capture image information inside the cavity and identify the cavity shape, texture and dynamic changes; The force sensor is used to monitor the contact force and direction between the catheter and the cavity wall to determine whether there is abnormal pressure; The ultrasonic sensor is used to obtain the depth information and physical characteristics of the cavity tissue; The inertial measurement unit is used to record the posture and motion trajectory of the catheter in the cavity and monitor the acceleration and angular velocity.

3. The digestive tract catheterization robot navigation method based on multimodal information fusion according to claim 1, characterized in that: In the step of preprocessing the collected multimodal data: Preprocessing methods include noise filtering, time synchronization and coordinate alignment.

4. The digestive tract catheterization robot navigation method based on multimodal information fusion according to claim 1, characterized in that: In the step of integrating multimodal data, extracting cavity environment features, and building a three-dimensional model of the human cavity in real time: The time features and motion dynamic features are extracted respectively, and the fused features are mapped into the three-dimensional geometric structure of the cavity through a three-dimensional modeling algorithm based on voxel representation, the cavity surface mesh is generated, and the three-dimensional model is output.

5. The digestive tract catheterization robot navigation method based on multimodal information fusion according to claim 1, characterized in that: In the step of analyzing the characteristics of the human cavity environment and generating local and global paths, the process of generating the local path is: For each neighbor node n of the current node n ′ , calculate its temporary cost g ′ (n ′ )=g(n)+c(n,n ′ );in c(n,n ′ ) is the cost of moving between nodes; Define inaccessible areas and dynamically adjust the route to avoid them; Determine changes in the environment ahead and make real-time adjustments.

6. The digestive tract catheterization robot navigation method based on multimodal information fusion according to claim 5, characterized in that: For each neighbor node n of the current node n ′ , calculate its temporary cost g ′ (n ′ )=g(n)+c(n,n ′ ) steps: If the neighbor node n′ is not in the open list and a better path is found, then f ′ (n ′ ) = g ′ (n ′ )+h(n ′ ), update g(n′) and f(n′), and set the parent node of n′ to n.

7. The digestive tract catheterization robot navigation method based on multimodal information fusion according to claim 5, characterized in that: For each neighbor node n of the current node n ′ , calculate its temporary cost g ′ (n ′ )=g(n)+c(n,n ′ ) steps: If the neighbor node n' is not in the open list, add it to the open list.

8. The digestive tract catheterization robot navigation method based on multimodal information fusion according to claim 1, characterized in that: In the step of analyzing the characteristics of the human cavity environment and generating local and global paths, the process of generating the global path is: The human cavity is represented as a weighted undirected graph G(V,E); where V represents the cavity node set, E represents the edge set connecting the nodes, and the edge weight ω(e) represents the travel cost between two nodes; Based on the overall human body cavity, the location of the target area, the size of the navigation device and the overall passability of the cavity are determined to generate a global path.

9. A digestive tract catheterization robot navigation system based on multimodal information fusion, applied to the digestive tract catheterization robot navigation method based on multimodal information fusion as claimed in claim 1, characterized in that: It includes data acquisition module, data processing module, 3D model building module, path generation module and data monitoring module; among which: The data acquisition module is used to collect real-time data of human cavities using a multimodal sensor; The data processing module is used to preprocess the collected multimodal data; wherein the preprocessing method includes noise filtering, time synchronization and coordinate alignment; The three-dimensional model building module is used to integrate multi-modal data, extract cavity environment characteristics, and build a three-dimensional model of the human cavity in real time; The path generation module is used to analyze the characteristics of the human body cavity environment and generate local and global paths; The data monitoring module is used to output the path to the tube placement robot controller, drive the tube placement robot, monitor the navigation abnormality status in real time, and collect and store the path, sensor status and operation data during the navigation process.

Images