Real-time localization and mapping methods and systems for medical mobile robots

Abstract

This invention provides a method, system, storage medium, and electronic device for real-time localization and mapping (SLAM) of medical mobile robots, relating to the field of SLAM technology. First, a monocular RGB camera, suitable for complex environments and with lower cost, is selected to capture video streams of real-world scenes. Then, a convolutional gated recurrent unit is used to estimate the depth information and camera pose corresponding to each keyframe image by utilizing the optical flow between different keyframe images. Next, the volume rendering capability of neural radiation fields is fused with the ability of hierarchical voxel meshes to capture global and local scene information. A hash encoding method, which requires less storage space and allows for easy searching of the corresponding mesh for sampling points, is used to quickly render larger scenes with limited resources. This not only improves localization and mapping accuracy but also enables medical mobile robots to adopt appropriate strategies for different terrain conditions, improving the safety and adaptability of medical mobile robots to complex terrain.

Description

Technical Field

[0001] This invention relates to the field of SLAM technology, and more specifically to a real-time localization and mapping method, system, storage medium, and electronic device for use in medical mobile robots. Background Technology

[0002] Today, Simultaneous Localization and Mapping (SLAM) technology is applied to a wide variety of scenarios, primarily to solve the localization and mapping problems of medical mobile robots when moving in unknown environments.

[0003] In related technologies, the paper "NICE-SLAM: Neural Implicit Scalable Encoding for SLAM" (Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys; Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 12786-12796) discloses a feasible scheme that uses an RGB-D camera as the image input sensor. Four feature grids and their corresponding decoders represent the scene geometry and appearance, thus dividing the scene into voxel grids. The viewing ray for each pixel is tracked using camera calibration estimated with the colmap tool. By sampling points along the viewing ray and finding the corresponding voxel cube grid, the features of the points are represented by the eight vertices of the grid cube using trilinear interpolation. These features are then input into four decoders to obtain occupancy values ​​from coarse to fine granular, and finally, the depth and color values ​​of the ray are rendered. By minimizing the re-rendering loss of depth and color, the depth and color values ​​of selected keyframes are optimized to display camera pose and scene geometry in an alternately optimized manner.

[0004] However, the aforementioned NICE-SLAM technology only takes video stream images and camera poses as input, and the training network has a long convergence time. If it is applied to physical robots, its real-time performance needs to be improved. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a method, system, storage medium, and electronic device for real-time localization and mapping of medical mobile robots, solving the technical problem of insufficient real-time performance.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A real-time localization and mapping method for medical mobile robots includes:

[0010] The medical mobile robot uses a built-in monocular RGB camera to capture video streams of real-world scenes and uses COLMAP to extract several keyframe images from the video stream to construct a keyframe list.

[0011] The keyframe images in the keyframe list are traversed, and the optical flow between different keyframe images is used by a convolutional gated recurrent unit to estimate the depth image, camera pose and inverse depth edge covariance corresponding to each keyframe image.

[0012] Based on the depth image, camera pose, and color information of each keyframe image, a coarse 3D map of the real-world scene is pre-constructed, and the coarse 3D map is divided into voxel grids of three different resolution sizes to obtain layered coarse-grained, medium-grained, and fine-grained scene maps; wherein the eight vertex features of each voxel grid are stored using hash encoding.

[0013] Based on the camera pose corresponding to the current keyframe image and each pixel on the image, sampling is performed using an expansion mask centered on the point of interest, and the voxel grid where each sampling point is located on a scene map of different granularities is searched by hash encoding.

[0014] Based on the multilayer perceptron corresponding to each voxel grid, the occupancy value and color of each sampling point are obtained, and volume rendering is used to generate RGB images and depth images;

[0015] Based on each keyframe image and its corresponding depth image, camera pose, rendered RGB image, and rendered depth image, a loss function is constructed by adding the depth uncertainty calculated based on the inverse depth edge covariance.

[0016] Minimize the loss function to determine the optimal parameters of each layer of the multilayer perceptron in order to render a real-time 3D map.

[0017] Preferably, the chassis of the medical mobile robot includes a retractable omnidirectional wheel assembly, and the real-time localization and mapping method further includes:

[0018] Based on the real-time 3D map, the road conditions ahead of the medical mobile robot in its current pose are determined, and the distance between the medical mobile robot and surrounding objects is estimated.

[0019] When the vertical difference in local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, the medical mobile robot is controlled to adaptively extend the omnidirectional wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, the omnidirectional wheel assembly remains retracted.

[0020] Preferably, the depth image, camera pose, and inverse depth edge covariance corresponding to the current keyframe image are estimated by utilizing the optical flow between different keyframe images through a convolutional gated recurrent unit; including:

[0021] The current keyframe image is passed through a convolutional layer to obtain the first feature map. The first feature map and the hidden state vector of the previous keyframe image are concatenated together and input into a gated recurrent unit. Based on the calculation result of the gated recurrent unit, the hidden state vector of the current keyframe image is obtained. Then, the image is passed through another convolutional layer to obtain the second feature map.

[0022] Based on the first feature map, the current optical flow measurement value is calculated; and by introducing a smoothness assumption, the current optical flow measurement value and its weights are updated by minimizing the energy function based on the second feature map.

[0023] Based on the current optical flow measurements and their weights, the inter-image reprojection error equations are linearized into a camera / depth arrow-shaped block sparse Hessian matrix.

[0024] The simplified camera matrix is ​​calculated using the Schur complement of the block sparse Hessian matrix, the simplified camera moments are decomposed by Cholesky, and the camera pose and depth image corresponding to the current keyframe image are solved by the previous iteration and the subsequent iteration.

[0025] Based on the camera pose and depth image, the inverse depth edge covariance and camera pose edge covariance corresponding to the current keyframe image are calculated respectively.

[0026] Preferably, the sampling method based on the expansion mask centered on the point of interest refers to:

[0027] Determine the color and brightness distribution of the current keyframe image, find the region with the most prominent color or the most significant brightness change in the image, take the center point of the region as the point of interest, and set the interval range as a square of a fixed number of pixels to perform iterative sampling as the region of interest.

[0028] Preferably, the occupancy value and color of each sampling point are obtained based on the multilayer perceptron corresponding to each voxel grid layer; represented as:

[0029]

[0030]

[0031] Among them, O P Indicates the occupancy value of point P; l represents the layer index of the voxel mesh; This represents the occupancy value of point P in the l-th voxel grid; P represents the three-dimensional coordinates of the sampling point. Let represent the feature vector of point P in the l-layer voxel grid, which is calculated by the difference of the feature lines of the eight vertices of the grid; θ represents the parameters of the multilayer perceptron; C P The color value of point P is represented by decoding using only a multilayer perceptron corresponding to a fine-grained resolution voxel grid; g β This is represented as a multilayer perceptron used for decoding color.

[0032] Preferably, the process of constructing the loss function specifically includes:

[0033] (1) Constructing the color loss:

[0034] L rgb (T,θ)=||II′(T,θ)|| 2

[0035] Where I′ represents the pixel color value of the rendered RGB image; I represents the pixel color value of the current keyframe image; T represents the camera pose; θ is the multilayer perceptron parameter; ||·|| 2 Represents the 2-norm;

[0036] (2) Constructing depth loss:

[0037]

[0038] Where D′ represents the rendered depth image; ∑ D This represents depth uncertainty and is used to supervise network learning.

[0039] (3) Constructing color consistency loss:

[0040]

[0041] For a pixel in the current keyframe image m, let it be denoted as r. m The number of pixels that the current keyframe image m projects onto its neighboring keyframe image n is denoted as r. m→n R represents the number of pixels sampled in the region of interest in each iteration; k m Represents a list of keyframes, excluding the current keyframe image m itself; I(·) represents the pixel color value; ||·||1 represents the 1-norm;

[0042] (4) Constructing optical flow loss

[0043]

[0044] Where GM(·) represents optical flow estimation.

[0045] (5) Construct the total loss function

[0046] L = α rgb L rgb (T,θ)+α D L D (T,θ)+α warp L warp +α flow L flow

[0047] Where, α rgb The weights representing the color loss, α D The weights α represent the depth loss. warp The weight α represents the color consistency loss. flow This represents the weight of optical flow loss.

[0048] A real-time localization and mapping system for medical mobile robots includes:

[0049] The extraction module is used to acquire video streams of real-world scenes through the monocular RGB camera built into the medical mobile robot, and to extract several key frame images from the video stream using COLMAP to construct a key frame list.

[0050] The localization module is used to traverse the keyframe images in the keyframe list and estimate the depth image, camera pose, and inverse depth edge covariance corresponding to each keyframe image by using the optical flow between different keyframe images through a convolutional gated recurrent unit.

[0051] The mapping module is used to pre-construct a coarse 3D map of the real-world scene based on the depth image, camera pose, and color information of each keyframe image, and to divide the coarse 3D map into voxel grids of three different resolution sizes to obtain layered coarse-grained, medium-grained, and fine-grained scene maps; wherein the eight vertex features of each voxel grid are stored using hash encoding.

[0052] Based on the camera pose corresponding to the current keyframe image and each pixel in the image, sampling is performed using an expansion mask centered on the point of interest, and the voxel grid where each sampling point is located on a scene map of different granularities is searched by hash encoding.

[0053] It is used to obtain the occupancy value and color of each sampling point based on the multilayer perceptron corresponding to each voxel grid, and to generate RGB image and depth image by volume rendering;

[0054] To construct a loss function, based on each keyframe image and its corresponding depth image, camera pose, rendered RGB image, and rendered depth image, add the depth uncertainty calculated based on the inverse depth edge covariance;

[0055] And to minimize the loss function, determine the optimal parameters of the multilayer perceptron for each layer to render a real-time 3D map.

[0056] Preferably, the chassis of the medical mobile robot includes a retractable omnidirectional wheel assembly, and the real-time positioning and mapping system further includes:

[0057] The control module is used to determine the road conditions ahead of the medical mobile robot in its current pose and to estimate the distance between the medical mobile robot and surrounding objects based on the real-time 3D map.

[0058] And when the vertical difference in local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, control the medical mobile robot to adaptively extend the omnidirectional wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, keep the omnidirectional wheel assembly retracted.

[0059] A storage medium characterized in that it stores a computer program for real-time localization and mapping applied to a medical mobile robot, wherein the computer program causes a computer to perform the real-time localization and mapping method as described above.

[0060] An electronic device, comprising:

[0061] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the instantaneous localization and mapping as described above.

[0062] (III) Beneficial Effects

[0063] This invention provides a method, system, storage medium, and electronic device for real-time localization and mapping of medical mobile robots. Compared with existing technologies, it has the following advantages:

[0064] This invention can be applied to, but is not limited to, medical mobile robots operating indoors. First, a monocular RGB camera, suitable for complex environments and with lower cost, is used to capture video streams of real-world scenes. Then, a convolutional gated recurrent unit is used to estimate the depth information and camera pose corresponding to each keyframe image by utilizing the optical flow between different keyframe images. Next, the volume rendering capability of neural radiation fields is fused with the ability of hierarchical voxel meshes to capture global and local scene information. A hash encoding method, which requires less storage space and allows for easy searching of the corresponding mesh for sampling points, is used to quickly render larger scenes with limited resources. Furthermore, depth supervision is added through uncertainty estimation to provide accurate information for the neural radiation fields used to construct scenes during movement, further accelerating convergence and enabling the rendering of high-precision, real-time 3D maps from keyframe images. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 A block diagram illustrating a real-time localization and mapping method for medical mobile robots, as provided in an embodiment of the present invention.

[0067] Figure 2 A block diagram illustrating another real-time localization and mapping method for medical mobile robots provided in an embodiment of the present invention;

[0068] Figure 3 A technical roadmap for a real-time localization and mapping method for medical mobile robots, provided by an embodiment of the present invention;

[0069] Figure 4 An example diagram of a medical mobile robot provided for an embodiment of the present invention (the universal wheel assembly is in a retracted state);

[0070] Figure 5 An example diagram of a medical mobile robot provided for an embodiment of the present invention (the universal wheel assembly is in an extended state);

[0071] Figure 6 A schematic diagram of an adaptive extension omnidirectional wheel assembly for a medical mobile robot descending a step, provided in an embodiment of the present invention;

[0072] Figure 7 This is a schematic diagram of an adaptive extension omnidirectional wheel assembly for a medical mobile robot, provided as an embodiment of the present invention, for climbing a step. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] This application provides a method, system, storage medium, and electronic device for real-time localization and mapping of medical mobile robots, which solves the technical problem of needing to improve real-time performance.

[0075] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0076] This invention first employs a convenient and low-power monocular RGB camera, which is more suitable for complex environments. It then uses a convolutional gated recurrent unit to estimate the camera pose and depth map using optical flow between multiple images. Simultaneously, hash encoding maps high-dimensional spaces or feature vectors to a low-dimensional hash table, significantly reducing computer storage requirements. A ray sampling method that reduces the number of samples combines the advantages of Neural Radiance Fields (NeRF) and hierarchical voxel grids to construct an accurate radiance field from the image stream. A novel loss function is designed that considers image color loss, camera pose loss, color consistency loss between adjacent pixels, and optical flow loss between frame pairs. Backpropagation minimizes the error between the rendered radiance field map and the ground truth map captured by the camera, outputting a near-perfect pose estimate and a dense depth map. Furthermore, uncertainty estimation adds depth supervision, providing accurate information for constructing the neural radiance field of the scene during movement, accelerating convergence and rendering a high-precision, real-time 3D map from the image.

[0077] Furthermore, the chassis of the medical mobile robot includes a retractable omnidirectional wheel assembly. Using the final rendered real-time 3D map, the estimated distance (i.e., depth) between surrounding objects and the medical mobile robot helps it determine the terrain in front of it at its current pose. If a preset slope exists in front of it and reaches a preset distance, the medical mobile robot automatically extends the omnidirectional wheel assembly, reaches a flat surface, and then retracts the omnidirectional wheel assembly, ensuring the robot's center of gravity is stable and aesthetically pleasing.

[0078] In summary, the method provided by the embodiments of the present invention can not only improve the accuracy of positioning and mapping, but also enable medical mobile robots to adopt corresponding strategies according to different terrain conditions, thereby improving the safety of medical mobile robots and their adaptability to complex terrain.

[0079] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0080] Example 1:

[0081] like Figure 1 As shown, a real-time localization and mapping method for medical mobile robots includes:

[0082] S1. The medical mobile robot uses a built-in monocular RGB camera to capture video streams of real-world scenes and uses COLMAP to extract several keyframe images from the video stream to construct a keyframe list.

[0083] S2. Traverse the keyframe images in the keyframe list, and use the optical flow between different keyframe images through the convolution gated loop unit to estimate the depth image, camera pose and inverse depth edge covariance corresponding to each keyframe image.

[0084] S3. Based on the depth image, camera pose, and color information of each keyframe image, a rough 3D map of the real-world scene is pre-constructed, and the rough 3D map is divided into voxel grids of three different resolution sizes to obtain layered coarse-grained, medium-grained, and fine-grained scene maps; wherein the eight vertex features of each voxel grid are stored using hash encoding.

[0085] S4. Based on the camera pose corresponding to the current keyframe image and each pixel on the image, sampling is performed using an expansion mask centered on the point of interest, and the voxel grid where each sampling point is located on a scene map of different granularities is searched by hash encoding.

[0086] S5. Based on the multilayer perceptron corresponding to each voxel grid, obtain the occupancy value and color of each sampling point, and generate an RGB image and a depth image through volume rendering;

[0087] S6. Based on each keyframe image and its corresponding depth image, camera pose, rendered RGB image, and rendered depth image, add the depth uncertainty calculated based on the inverse depth edge covariance to construct a loss function.

[0088] S7. Minimize the loss function to determine the optimal parameters of each layer of the multilayer perceptron in order to render a real-time 3D map.

[0089] The embodiments of the present invention can be applied to, but are not limited to, medical mobile robots for indoor activities. A monocular RGB camera is used for real-time localization and mapping, so as to render a high-precision and real-time 3D map from key frame images under limited resources.

[0090] In an alternative implementation, the chassis of the medical mobile robot includes a retractable omnidirectional wheel assembly, such as... Figure 2 As shown, the instant positioning and map building method further includes:

[0091] S8. Based on the real-time 3D map, determine the road conditions ahead of the medical mobile robot in its current pose, and estimate the distance between the medical mobile robot and surrounding objects.

[0092] When the vertical difference in local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, the medical mobile robot is controlled to adaptively extend the omnidirectional wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, the omnidirectional wheel assembly remains retracted.

[0093] In this embodiment of the invention, the medical mobile robot can be controlled to adopt corresponding strategies for different terrain conditions, thereby improving the safety of the medical mobile robot and its adaptability to complex terrain.

[0094] like Figure 3 As shown, Figure 3 A technical roadmap for the real-time positioning and map building method provided in the embodiments of the present invention is given. The following will combine... Figure 3 The steps of the above scheme are described in detail:

[0095] In step S1, the video stream of the real-world scene is acquired through the monocular RGB camera built into the medical mobile robot, and several key frame images are extracted from the video stream using COLMAP to construct a key frame list.

[0096] It should be noted that many current robotic 3D reconstruction solutions are based on RGB-D or LiDAR sensors, but scene reconstruction using only monocular RGB images offers a more convenient solution. RGB-D cameras are prone to malfunction in poor lighting conditions, and LiDAR is heavier and more expensive than monocular cameras, and its light scattering and reflection caused by materials such as glass can lead to inaccurate positioning and estimation. Prolonged use on mobile devices may also result in positional shifts.

[0097] Furthermore, while stereo cameras simplify depth estimation to a one-dimensional parallax search, they are highly dependent on the camera's calibration accuracy, which can easily lead to calibration errors in practice. In contrast, monocular RGB cameras (cameras) are inexpensive, lightweight, and consume less power, extending device battery life. They also represent the simplest sensor configuration for calibration, making data processing much easier.

[0098] Accordingly, in this embodiment of the invention, after conducting a physical survey of the terrain in a real application scenario, a monocular RGB camera is built into the medical mobile robot to collect video streams of the real-world scene. Then, using the open-source dynamic structure package tool COLMAP, several keyframe images are extracted from the video stream to construct a keyframe list.

[0099] After obtaining the list of keyframes, this embodiment of the invention then performs a positioning operation, as detailed in step S2.

[0100] In step S2, the keyframe images in the keyframe list are traversed, and the depth image, camera pose, and inverse depth edge covariance corresponding to each keyframe image are estimated by using the optical flow between different keyframe images through a convolutional gated recurrent unit.

[0101] This invention overcomes the limitation of monocular RGB cameras in estimating surrounding depth maps by designing a Convolutional Gated Recurrent Unit (ConvGRU) between consecutive keyframes. The ConvGRU consists of a gated recurrent unit (GRU) and a convolutional neural network (CNN). Specifically, it replaces the fully connected layers in the GRU with convolutional layers, thereby enabling the capture of spatial transformations in spatiotemporal sequence data.

[0102] Accordingly, the depth image, camera pose, and inverse depth edge covariance corresponding to the current keyframe image are estimated by utilizing the optical flow between different keyframe images through a convolutional gated recurrent unit; specifically including:

[0103] The current keyframe image is passed through a convolutional layer to obtain the first feature map. The first feature map and the hidden state vector of the previous keyframe image are concatenated together and input into a gated recurrent unit. Based on the calculation result of the gated recurrent unit, the hidden state vector of the current keyframe image is obtained. Then, the image is passed through another convolutional layer to obtain the second feature map.

[0104] Based on the first feature map, calculate the current optical flow measurement value p. ij Furthermore, a smoothness assumption is introduced, and based on the second feature map, the current optical flow measurement p is updated by minimizing the energy function. ij and its weight ∑p ij .

[0105] Based on the current optical flow measurements and their weights, the inter-image reprojection error equations are linearized into a camera / depth arrow-shaped block sparse Hessian matrix H∈R. (c+p)×(c+p) Where c and p are the dimensions of the camera and the point, respectively, the linearized system of equations is Hx = b, specifically expressed as:

[0106]

[0107] The above formulas are the specific parameters of the equation system normal form, where C represents the block camera matrix (covering camera intrinsics and pixel coordinates), E represents the identity matrix, the superscript T indicates transpose, A represents the diagonal matrix of inverse depth, Δɑ represents the increment of the camera pose Lie algebra, and ΔD... -1 represents the inverse depth increment for each pixel, and b represents the constant term consisting of the depth residual m and the pose residual m.

[0108] To solve the linear least squares problem, the simplified camera matrix H is calculated using the Schur complement of the block sparse Hessian matrix. T For the simplified camera moment H T =LL T Cholesky decomposition is performed, and the camera pose T corresponding to the current keyframe image is solved through previous and subsequent iterations. Based on the solved camera pose T, the depth image D is further obtained.

[0109] To make good use of the output (T, D) of the above dense mapping, this embodiment of the invention further calculates the inverse depth D. -1 Edge covariance and camera pose T edge covariance quantify the uncertainty of depth estimation, camera position and orientation estimation for each pixel. This adds useful depth supervision to training, further optimizing or fusing sensor data and accelerating convergence as more information is provided. The formula for edge covariance is shown below:

[0110] ∑ D-1 =A -1 +A -T E T ∑ T EA -1

[0111] ∑ T =(LL T ) -1

[0112] Where L represents a triangular matrix.

[0113] After completing the positioning operation, the embodiment of the present invention then performs the mapping operation, as detailed in steps S3 to S7.

[0114] In step S3, based on the depth image, camera pose, and color information of each keyframe image, a rough 3D map of the real-world scene is pre-constructed, and the rough 3D map is divided into voxel grids of three different resolution sizes to obtain layered coarse-grained, medium-grained, and fine-grained scene maps; wherein the eight vertex features of each voxel grid are stored using hash encoding.

[0115] For example, in this step, the coarse 3D map is divided into grids of three different resolution sizes. The coarse-grained resolution grid learns information about the overall scene, and the grid size is set to a cube with a side length of 2m to effectively simulate coarse scene geometry (such as walls and floors). The medium-grained and fine-grained resolution grids learn fine details, and the grid sizes are set to a side length of 16cm and 8cm, respectively, to obtain high-frequency geometric details of the scene. Each grid stores the features of eight vertices of the grid based on hash encoding.

[0116] In step S4, based on the camera pose corresponding to the current keyframe image and each pixel on the image, sampling is performed using an expansion mask centered on the point of interest, and the voxel grid where each sampling point is located on different granularity scene maps is searched by hash encoding.

[0117] After obtaining the camera pose, multiple rays can be formed based on the camera center and each pixel of any keyframe image. Sampling is carried out using an expansion mask centered on the interest point. The voxel grid where the sampling point is located is searched by hash encoding, and the feature vector of the sampling point in the voxel grid is calculated by trilinear interpolation using the features of the eight vertices of the grid.

[0118] It should be noted that the aforementioned interest-based expansion mask method specifically refers to:

[0119] Determine the color and brightness distribution of the current keyframe image, find the region with the most prominent color or the region with the most significant brightness change, take the center point of the region as the point of interest, and set the interval range as a square of a fixed number of pixels (e.g., 5 pixels) to perform iterative sampling as the region of interest.

[0120] Understandably, in this embodiment of the invention, the neural radiation field implicitly simulates light propagation to render and reconstruct the scene. It also captures scene information of different dimensions using grids of varying resolutions, incorporates uncertainty estimation as supervision, and employs region-of-interest (ROI) sampling—a strategy that relaxes the sampling of interest points. Sampling is performed from an expanded mask centered on the interest point. After obtaining the interest point, the sampling region is iterated multiple times using a 5×5 expansion method. Both of these methods shorten the training time, allow the network to converge faster, and improve the real-time performance of simultaneous mapping and localization.

[0121] In step S5, based on the multilayer perceptron corresponding to each voxel grid, the occupancy value and color of each sampling point are obtained, and volume rendering is used to generate RGB images and depth images.

[0122] In this embodiment of the invention, each voxel mesh is trained using a multilayer perceptron. Accordingly, this step specifically includes:

[0123]

[0124] Among them, O P Indicates the occupancy value of point P; l represents the layer index of the voxel mesh; This represents the occupancy value of point P in the l-th voxel grid; P represents the three-dimensional coordinates of the sampling point. Let represent the feature vector of point P in the l-layer voxel grid, which is calculated by the difference of the feature lines of the eight vertices of the grid; θ represents the parameters of the multilayer perceptron; C P The color value of point P is represented by decoding using only a multilayer perceptron corresponding to a fine-grained resolution voxel grid; g β This is represented as a multilayer perceptron used for decoding color.

[0125] The above describes the forward propagation process. In this embodiment of the invention, a loss function is also required for backpropagation to alternately update pose and depth. The loss is calculated using the rendered image, the entity image and color image captured by the camera, and the estimated depth and pose image. The loss function is constructed and minimized. For details, please refer to steps S6 to S7.

[0126] In step S6, based on each keyframe image and its corresponding depth image, camera pose, rendered RGB image, and rendered depth image, depth uncertainty calculated based on the inverse depth edge covariance is added to construct a loss function; specifically including:

[0127] (1) Constructing the color loss:

[0128] L rgb (T,θ)=||II′(T,θ)|| 2

[0129] Where I′ represents the pixel color value of the rendered RGB image; I represents the pixel color value of the current keyframe image; T represents the camera pose; θ is the multilayer perceptron parameter; ||·|| 2 Represents the 2-norm;

[0130] (2) Constructing depth loss:

[0131]

[0132] Where D′ represents the rendered depth image; ∑ D This represents depth uncertainty and is used to supervise network learning.

[0133] (3) Constructing color consistency loss:

[0134]

[0135] For a pixel in the current keyframe image m, let it be denoted as r. mThe number of pixels that the current keyframe image m projects onto its neighboring keyframe image n is denoted as r. m→n R represents the number of pixels sampled in the region of interest in each iteration; k m Represents a list of keyframes, excluding the current keyframe image m itself; I(·) represents the pixel color value; ||·||1 represents the 1-norm;

[0136] (4) Constructing optical flow loss

[0137]

[0138] Where GM(·) represents optical flow estimation.

[0139] (5) Construct the total loss function

[0140] L = α rgb L rgb (T,θ)+α D L D (T,θ)+α warp L warp +α flow L flow

[0141] Where, α rgb The weights representing the color loss, α D The weights α represent the depth loss. warp The weight α represents the color consistency loss. flow This represents the weight of optical flow loss.

[0142] In step S7, the loss function is minimized to determine the optimal parameters of each layer of the multilayer perceptron in order to render a real-time 3D map.

[0143] Thus, this embodiment of the invention achieves the rendering of a high-precision, real-time 3D map from keyframe images under limited resources. Furthermore, based on this real-time 3D map, this embodiment of the invention also provides solutions for the navigation and motion control of medical mobile robots, as detailed in step S8.

[0144] In step S8, based on the real-time 3D map, the road conditions ahead of the medical mobile robot in its current pose are determined, and the distance between the medical mobile robot and surrounding objects is estimated. When the vertical difference in local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, the medical mobile robot is controlled to adaptively extend the omnidirectional wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, the omnidirectional wheel assembly remains retracted.

[0145] For example, such as Figures 4-5As shown, the medical mobile robot provided in this embodiment of the invention adopts a humanoid appearance, and the chassis of the medical mobile robot includes a retractable universal wheel assembly to cope with sloping, jumping terrain such as going down stairs.

[0146] Specifically, when the vertical difference in local terrain is greater than or equal to a preset first threshold of 0-4 cm and the distance is less than or equal to a second threshold of 30 cm, the medical mobile robot is controlled to adaptively extend the omnidirectional wheel assembly to lower the center of gravity of the medical mobile robot. Figures 6-7 As shown; otherwise, the omnidirectional wheel assembly remains retracted. This improves the safety and adaptability of the medical mobile robot to complex terrain.

[0147] The following are specific application examples of embodiments of the present invention:

[0148] Data used: Maps were created in the ICU of Hospital A, the Infectious Diseases Department of Hospital B, and the ICU of Hospital C (with some slope and thresholds for each scenario).

[0149] Specific settings: Select K=16 frames, where 5 frames are randomly selected from the keyframe list, 10 frames are randomly selected from the most recent 20 keyframes, and the current frame is also selected. Set the batch size to 8, divide the region of interest sampling into 5*5 pixels, and the final L=L rgb (T,θ)+0.1L D (T,θ)+0.5L warp +0.001L flow When the robot detects a vertical difference in local terrain of 0-4cm, it will adaptively extend the omnidirectional wheel assembly when the distance approaches 30cm; otherwise, it will remain in the retracted state.

[0150] Example 2:

[0151] This invention provides a real-time localization and mapping system for medical mobile robots, comprising:

[0152] The extraction module is used to acquire video streams of real-world scenes through the monocular RGB camera built into the medical mobile robot, and to extract several key frame images from the video stream using COLMAP to construct a key frame list.

[0153] The localization module is used to traverse the keyframe images in the keyframe list and estimate the depth image, camera pose, and inverse depth edge covariance corresponding to each keyframe image by using the optical flow between different keyframe images through a convolutional gated recurrent unit.

[0154] The mapping module is used to pre-construct a coarse 3D map of the real-world scene based on the depth image, camera pose, and color information of each keyframe image, and to divide the coarse 3D map into voxel grids of three different resolution sizes to obtain layered coarse-grained, medium-grained, and fine-grained scene maps; wherein the eight vertex features of each voxel grid are stored using hash encoding.

[0155] Based on the camera pose corresponding to the current keyframe image and each pixel in the image, sampling is performed using an expansion mask centered on the point of interest, and the voxel grid where each sampling point is located on a scene map of different granularities is searched by hash encoding.

[0156] It is used to obtain the occupancy value and color of each sampling point based on the multilayer perceptron corresponding to each voxel grid, and to generate RGB image and depth image by volume rendering;

[0157] To construct a loss function, based on each keyframe image and its corresponding depth image, camera pose, rendered RGB image, and rendered depth image, add the depth uncertainty calculated based on the inverse depth edge covariance;

[0158] And to minimize the loss function, determine the optimal parameters of the multilayer perceptron for each layer to render a real-time 3D map.

[0159] In an optional implementation, the chassis of the medical mobile robot includes a retractable omnidirectional wheel assembly, and the real-time localization and mapping system further includes:

[0160] The control module is used to determine the road conditions ahead of the medical mobile robot in its current pose and to estimate the distance between the medical mobile robot and surrounding objects based on the real-time 3D map.

[0161] And when the vertical difference in local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, control the medical mobile robot to adaptively extend the omnidirectional wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, keep the omnidirectional wheel assembly retracted.

[0162] Example 3:

[0163] This invention provides a storage medium storing a computer program for real-time localization and mapping (RTM) of a medical mobile robot, wherein the computer program causes a computer to execute the RTM method as described in Embodiment 1.

[0164] Example 4:

[0165] This invention provides an electronic device, comprising:

[0166] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the real-time localization and mapping method as described in Example 1.

[0167] It is understood that the real-time localization and mapping system, storage medium and electronic device for medical mobile robots provided in the embodiments of the present invention correspond to the real-time localization and mapping method for medical mobile robots provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the real-time localization and mapping method, and will not be repeated here.

[0168] In summary, compared with existing technologies, it has the following beneficial effects:

[0169] 1. The embodiments of the present invention conduct a physical survey of the terrain and topography in real application scenarios, consider selecting the most suitable sensor, and overcome the drawback of monocular RGB cameras being unable to estimate the surrounding depth map by designing a convolutional gating loop unit between consecutive key frames.

[0170] 2. The embodiments of the present invention integrate the volume rendering capability of neural radiation fields with the ability of layered voxel meshes to capture global and local scene information, and use hash encoding, which has a smaller storage space and can find the corresponding mesh of the sampling point more quickly, to render larger scenes with faster convergence speed and improve real-time performance under limited resources.

[0171] 3. In order to accelerate the training speed, this embodiment of the invention first uses the edge covariance of the output of the convolutional gated recurrent unit as an uncertainty estimate, and incorporates this information as the input of the multilayer perceptron, which converges faster. In addition, the sampling method is improved. Instead of random sampling based on the original neural radiation field, it is based on sampling of the region of interest of the interest point and sampling in the expanded mask centered on the interest point.

[0172] 4. The embodiments of the present invention set a brand-new loss function, which takes into account the smoothness between images and pixels, resulting in a lower error rate in the rendered image, smaller geometric errors and scene object color errors, and more accurate depth estimation, so that the universal wheel assembly can be extended or retracted at a more accurate time.

[0173] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0174] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time localization and mapping method for medical mobile robots, characterized in that, include: The medical mobile robot uses a built-in monocular RGB camera to capture video streams of real-world scenes and uses COLMAP to extract several keyframe images from the video stream to construct a keyframe list. The keyframe images in the keyframe list are traversed, and the optical flow between different keyframe images is used by a convolutional gated recurrent unit to estimate the depth image, camera pose and inverse depth edge covariance corresponding to each keyframe image. Based on the depth image, camera pose, and color information of each keyframe image, a coarse 3D map of the real-world scene is pre-constructed, and the coarse 3D map is divided into voxel grids of three different resolution sizes to obtain layered coarse-grained, medium-grained, and fine-grained scene maps; wherein the eight vertex features of each voxel grid are stored using hash encoding. Based on the camera pose corresponding to the current keyframe image and each pixel on the image, sampling is performed using an expansion mask centered on the point of interest, and the voxel grid where each sampling point is located on a scene map of different granularities is searched by hash encoding. Based on the multilayer perceptron corresponding to each voxel grid, the occupancy value and color of each sampling point are obtained, and volume rendering is used to generate RGB images and depth images; Based on each keyframe image and its corresponding depth image, camera pose, rendered RGB image, and rendered depth image, a loss function is constructed by adding the depth uncertainty calculated based on the inverse depth edge covariance. Minimize the loss function to determine the optimal parameters of each layer of the multilayer perceptron in order to render a real-time 3D map; The sampling method based on the expansion mask centered on the point of interest refers to: Determine the color and brightness distribution of the current keyframe image, find the region with the most prominent color or the region with the most significant brightness change in the image, take the center point of the region as the point of interest, and set the interval range as a square of a fixed number of pixels to perform iterative sampling as the region of interest; Methods for obtaining inverse depth edge covariance include: Based on the camera pose and depth image, the inverse depth edge covariance and camera pose edge covariance corresponding to the current keyframe image are calculated respectively. The formula for marginal covariance is shown below: ∑ T =(LL T ) -1 Where L represents a triangular matrix; Represents the inverse depth edge covariance, ∑ T Let A represent the edge covariance of camera pose; let A represent the diagonal matrix of inverse depth; let E represent the identity matrix; and let the superscript T represent the transpose.

2. The real-time positioning and map building method as described in claim 1, characterized in that, The chassis of the medical mobile robot includes a retractable omnidirectional wheel assembly, and the real-time localization and mapping method further includes: Based on the real-time 3D map, the road conditions ahead of the medical mobile robot in its current pose are determined, and the distance between the medical mobile robot and surrounding objects is estimated. When the vertical difference in local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, the medical mobile robot is controlled to adaptively extend the omnidirectional wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, the omnidirectional wheel assembly remains retracted.

3. The real-time positioning and map building method as described in claim 1, characterized in that, The optical flow between different keyframe images is used by a convolutional gated recurrent unit to estimate the depth image, camera pose, and inverse depth edge covariance corresponding to the current keyframe image; including: The current keyframe image is passed through a convolutional layer to obtain the first feature map. The first feature map and the hidden state vector of the previous keyframe image are concatenated together and input into a gated recurrent unit. Based on the calculation result of the gated recurrent unit, the hidden state vector of the current keyframe image is obtained. Then, the image is passed through another convolutional layer to obtain the second feature map. Based on the first feature map, the current optical flow measurement value is calculated; and by introducing a smoothness assumption, the current optical flow measurement value and its weights are updated by minimizing the energy function based on the second feature map. Based on the current optical flow measurements and their weights, the inter-image reprojection error equations are linearized into a camera / depth arrow-shaped block sparse Hessian matrix. The simplified camera matrix is ​​calculated using the Schur complement of the block sparse Hessian matrix, the simplified camera moments are decomposed by Cholesky, and the camera pose and depth image corresponding to the current keyframe image are solved by the previous iteration and the subsequent iteration. Based on the camera pose and depth image, the inverse depth edge covariance and camera pose edge covariance corresponding to the current keyframe image are calculated respectively.

4. The real-time positioning and map building method as described in claim 1, characterized in that, The multilayer perceptron, based on each voxel grid layer, acquires the occupancy value and color of each sampling point; represented as: Among them, O P Indicates the occupancy value of point P; l represents the layer index of the voxel mesh; This represents the occupancy value of point P in the l-th voxel grid; P represents the three-dimensional coordinates of the sampling point. Let represent the feature vector of point P in the l-layer voxel grid, which is calculated by the trilinear difference of the eight vertices of the grid; θ represents the parameters of the multilayer perceptron; C P The color value of point P is represented by decoding using only a multilayer perceptron corresponding to a fine-grained resolution voxel grid; g β This is represented as a multilayer perceptron used for decoding color.

5. The real-time positioning and mapping method as described in any one of claims 1 to 4, characterized in that, The process of constructing the loss function specifically includes: (1) Constructing the color loss: L rgb (T,θ)=||I-I′(T,θ)|| 2 Where I′ represents the pixel color value of the rendered RGB image; I represents the pixel color value of the current keyframe image; T represents the camera pose; θ is the multilayer perceptron parameter; ||·|| 2 Represents the 2-norm; (2) Constructing depth loss: Where D′ represents the rendered depth image; ∑ D This represents depth uncertainty and is used to supervise network learning. (3) Constructing color consistency loss: For a pixel in the current keyframe image m, let it be denoted as r. m The number of pixels that the current keyframe image m projects onto its neighboring keyframe image n is denoted as r. m→n R represents the number of pixels sampled in the region of interest in each iteration; k m Represents a list of keyframes, excluding the current keyframe image m itself; I(·) represents the pixel color value; ||·||1 represents the 1-norm; (4) Constructing optical flow loss Where GM(·) represents optical flow estimation; (5) Construct the total loss function L=a rgb L rgb (T,θ)+α D L D (T,θ)+α warp L warp +a flow L flow Where, α rgb The weights representing the color loss, α D The weights α represent the depth loss. warp The weight α represents the color consistency loss. flow This represents the weight of optical flow loss.

6. A real-time localization and mapping system for medical mobile robots, characterized in that, include: The extraction module is used to acquire video streams of real-world scenes through the monocular RGB camera built into the medical mobile robot, and to extract several key frame images from the video stream using COLMAP to construct a key frame list. The localization module is used to traverse the keyframe images in the keyframe list and estimate the depth image, camera pose, and inverse depth edge covariance corresponding to each keyframe image by using the optical flow between different keyframe images through a convolutional gated recurrent unit. The mapping module is used to pre-construct a coarse 3D map of the real-world scene based on the depth image, camera pose, and color information of each keyframe image, and to divide the coarse 3D map into voxel grids of three different resolution sizes to obtain layered coarse-grained, medium-grained, and fine-grained scene maps; wherein the eight vertex features of each voxel grid are stored using hash encoding. Based on the camera pose corresponding to the current keyframe image and each pixel in the image, sampling is performed using an expansion mask centered on the point of interest, and the voxel grid where each sampling point is located on a scene map of different granularities is searched by hash encoding. It is used to obtain the occupancy value and color of each sampling point based on the multilayer perceptron corresponding to each voxel grid, and to generate RGB image and depth image by volume rendering; To construct a loss function by adding the depth uncertainty calculated based on the inverse depth edge covariance to each keyframe image and its corresponding depth image, camera pose, rendered RGB image, and rendered depth image; And to minimize the loss function, determine the optimal parameters of each layer of the multilayer perceptron, so as to render a real-time 3D map; The sampling method based on the expansion mask centered on the point of interest refers to: Determine the color and brightness distribution of the current keyframe image, find the region with the most prominent color or the region with the most significant brightness change in the image, take the center point of the region as the point of interest, and set the interval range as a square of a fixed number of pixels to perform iterative sampling as the region of interest; Methods for obtaining inverse depth edge covariance include: Based on the camera pose and depth image, the inverse depth edge covariance and camera pose edge covariance corresponding to the current keyframe image are calculated respectively. The formula for marginal covariance is shown below: ∑ T =(LL T ) -1 Where L represents a triangular matrix; Represents the inverse depth edge covariance, ∑ T Let A represent the edge covariance of camera pose; let A represent the diagonal matrix of inverse depth; let E represent the identity matrix; and let the superscript T represent the transpose.

7. The real-time positioning and mapping system as described in claim 6, characterized in that, The chassis of the medical mobile robot includes a retractable omnidirectional wheel assembly, and the real-time positioning and mapping system further includes: The control module is used to determine the road conditions ahead of the medical mobile robot in its current pose and to estimate the distance between the medical mobile robot and surrounding objects based on the real-time 3D map. And when the vertical difference in local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, control the medical mobile robot to adaptively extend the omnidirectional wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, keep the omnidirectional wheel assembly retracted.

8. A storage medium, characterized in that, It stores a computer program for real-time localization and mapping applied to medical mobile robots, wherein the computer program causes the computer to perform the real-time localization and mapping method as described in any one of claims 1 to 5.

9. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the real-time localization and mapping method as described in any one of claims 1 to 5.

Images