Instant positioning and mapping method and system applied to medical mobile robot

Abstract

The invention provides an instant positioning and mapping method and system applied to a medical mobile robot, a storage medium and electronic equipment, and relates to the technical field of SLAM. According to the method, firstly, a monocular RGB camera which is suitable for a complex environment and lower in cost is selected to collect video streams of a real world scene, and depth information and camera poses corresponding to each key frame image are estimated through a convolution gating circulation unit by means of optical streams between different key frame images. And then, fusing the volume rendering capability of the neural radiation field and the capability of capturing global and local scene information of the layered voxel grid, and realizing rapid rendering of a larger scene under the condition of limited resources by using a hash coding mode which has a small storage space and is easy to search for a grid corresponding to a sampling point. The positioning and mapping precision can be improved, the medical mobile robot can adopt corresponding strategies according to different ground conditions, and the safety of the medical mobile robot and the adaptability of the medical mobile robot to complex terrains are improved.

Description

Technical Field

[0001] The present invention relates to the field of SLAM technology, and in particular to a method, system, storage medium and electronic equipment for real-time positioning and map construction of a medical mobile robot. Background Art

[0002] Nowadays, Simultaneous Localization and Mapping (SLAM) technology is applied to a variety of scenarios, mainly to solve the positioning and mapping problems of medical mobile robots when moving in unknown environments.

[0003] In the related art, the paper "NICE-SLAM: Neural Implicit Scalable Encoding for SLAM" (Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys; Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 12786-12796) discloses a feasible solution, which uses an RGB-D camera as a sensor for image input, and four feature grids and their corresponding decoders represent the scene geometry and appearance, whereby the scene is divided into voxel grids one by one, and the camera calibration estimated by the colmap tool is used to track the viewing light of each pixel. By following the sampling points on the observation light and finding the corresponding voxel cube grid, the features of the points are represented by the eight vertices of the grid cube by trilinear interpolation, which are then input into four decoders to obtain occupancy values ​​from coarse to fine granularity, and then the depth and color values ​​of the light are rendered. Optimizes the depth and color values ​​of selected keyframes in a trade-off between optimizing the camera pose and scene geometry by minimizing the re-rendering loss of depth and color.

[0004] However, the above-mentioned NICE-SLAM technology only inputs video stream images and camera poses, and the training network convergence time is relatively long. If it is transplanted to a physical robot, the real-time performance needs to be improved. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a method, system, storage medium and electronic device for real-time positioning and map construction applied to a medical mobile robot, which solves the technical problem that real-time performance needs to be improved.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A real-time positioning and map building method for a medical mobile robot, comprising:

[0010] The medical mobile robot is equipped with a built-in monocular RGB camera to collect a video stream of a real-world scene, and a plurality of key frame images are extracted from the video stream using COLMAP to construct a key frame list;

[0011] Traversing the key frame images in the key frame list, and using the optical flow between different key frame images through a convolutional gated recurrent unit to estimate the depth image, camera pose, and inverse depth edge covariance corresponding to each key frame image;

[0012] Based on the depth image, camera pose and color information of each of the key frame images, a rough 3D map of the real world scene is pre-constructed, and the rough 3D map is divided into voxel grids according to three different resolutions to obtain hierarchical coarse-grained, medium-grained and fine-grained scene graphs; wherein each of the voxel grids uses hash coding to store eight vertex features of the grid;

[0013] Based on the camera pose corresponding to the current keyframe image and each pixel on its image, sampling is performed using a dilation mask centered on the interest point, and the voxel grid where each sampling point is located on the scene graph of different granularities is searched through hash coding;

[0014] Based on the multi-layer perceptron corresponding to each layer of the voxel grid, the occupancy value and color of each sampling point are obtained, and the RGB image and the depth image are generated by volume rendering;

[0015] Based on each of the key frame images and their corresponding depth images, camera poses, rendered RGB images, and rendered depth images, adding the depth uncertainty calculated based on the inverse depth edge covariance, and constructing a loss function;

[0016] The loss function is minimized to determine the optimal parameters of each layer of the multilayer perceptron to render a real-time 3D map.

[0017] Preferably, the chassis of the medical mobile robot includes a retractable universal wheel assembly, and the instant positioning and map building method further includes:

[0018] Based on the real-time 3D map, determine the road condition ahead of the medical mobile robot at the current position, and estimate the distance between the medical mobile robot and surrounding objects;

[0019] When the vertical direction difference of the local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, the medical mobile robot is controlled to adaptively extend the universal wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, the universal wheel assembly is kept retracted.

[0020] Preferably, the depth image, camera pose and inverse depth edge covariance corresponding to the current key frame image are estimated by using the optical flow between different key frame images through a convolution gated recurrent unit; including:

[0021] The current key frame image is passed through a convolution layer to obtain a first feature map, the first feature map and the hidden state vector of the previous key frame image are concatenated together, input into a gated recurrent unit, and according to the calculation result of the gated recurrent unit, the hidden state vector of the current key frame image is obtained, and then passed through a convolution layer to obtain a second feature map;

[0022] Based on the first feature map, calculate the current optical flow measurement value; and introduce a smoothness assumption, based on the second feature map, update the current optical flow measurement value and its weight by minimizing the energy function;

[0023] Based on the current optical flow measurement and its weight, the inter-image reprojection error equations are linearized into a block sparse Hessian matrix of camera / depth arrows;

[0024] The simplified camera matrix is ​​calculated using the Schur complement of the block sparse Hessian matrix, the simplified camera moment is decomposed by Cholesky, and the camera pose and depth image corresponding to the current key frame image are solved through the previous iteration and the next iteration;

[0025] Based on the camera pose and the depth image, the inverse depth edge covariance and the camera pose edge covariance corresponding to the current key frame image are calculated respectively.

[0026] Preferably, the sampling method based on the expansion mask centered on the point of interest refers to:

[0027] Determine the color and brightness distribution of the current keyframe image, find the area with the most prominent color or the most significant brightness change in the image, use the center point of the area as the point of interest, and set the interval range to a square with a fixed number of pixels as the area of ​​interest for iterative sampling.

[0028] Preferably, the multi-layer perceptron corresponding to each layer of the voxel grid is used to obtain the occupancy value and color of each sampling point; it is expressed as:

[0029]

[0030]

[0031] Among them, O P represents the occupancy value of point P; l represents the layer index of the voxel grid; represents the occupancy value of point P in the voxel grid of the lth layer; P represents the three-dimensional coordinates of the sampling point; represents the feature vector of point P in the l-layer voxel grid, which is calculated by the three-line difference of the eight vertex features of the grid; θ represents the parameters of the multi-layer perceptron; C P represents the color value of point P, which is decoded only by the multi-layer perceptron corresponding to the fine-grained resolution voxel grid; g β Represented as a multilayer perceptron for decoding color.

[0032] Preferably, the process of constructing the loss function specifically includes:

[0033] (1) Constructing color loss:

[0034] L rgb (T,θ)=||II′(T,θ)|| 2

[0035] Where I′ represents the pixel color value of the rendered RGB image; I represents the pixel color value of the current key frame image; T represents the camera pose; θ is the multi-layer perceptron parameter; ||·|| 2 represents the 2-norm;

[0036] (2) Constructing depth loss:

[0037]

[0038] Where D′ represents the rendered depth image; ∑ D Represents deep uncertainty, used for supervised network learning;

[0039] (3) Constructing color consistency loss:

[0040]

[0041] Among them, for the pixel in the current key frame image m, it is recorded as r m ; The pixel projected by the current key frame image m in the adjacent key frame image n is denoted as r m→n ; R represents the pixels sampled in the region of interest at each iteration; k m represents the key frame list, excluding the current key frame image m itself; I(·) represents the pixel color value; ||·||1 represents the 1 norm;

[0042] (4) Constructing optical flow loss

[0043]

[0044] Here, GM(·) represents the optical flow estimation.

[0045] (5) Constructing the total loss function

[0046] L=α rgb L rgb (T,θ)+α D L D (T,θ)+α warp L warp +α flow L flow

[0047] Among them, α rgb Represents the weight of color loss, α D represents the weight of the depth loss, α warp represents the weight of color consistency loss, α flow Represents the weight of the optical flow loss.

[0048] A real-time positioning and map building system for a medical mobile robot, comprising:

[0049] An extraction module, used to collect a video stream of a real-world scene through a monocular RGB camera built into the medical mobile robot, and extract a number of key frame images from the video stream using COLMAP to construct a key frame list;

[0050] A positioning module, used to traverse the key frame images in the key frame list, and estimate the depth image, camera pose and inverse depth edge covariance corresponding to each key frame image by using the optical flow between different key frame images through a convolutional gated recurrent unit;

[0051] A mapping module, for pre-building a rough 3D map of the real world scene based on the depth image, camera pose and color information of each of the key frame images, and dividing the rough 3D map into voxel grids according to three different resolutions to obtain hierarchical coarse-grained, medium-grained and fine-grained scene graphs; wherein each of the voxel grids uses hash coding to store eight vertex features of the grid;

[0052] It is used to sample the camera pose corresponding to the current key frame image and each pixel on its image by using a dilation mask centered on the point of interest, and search the voxel grid where each sampling point is located on the scene graph of different granularities through hash coding;

[0053] Used to obtain the occupancy value and color of each sampling point based on the multi-layer perceptron corresponding to each layer of the voxel grid, and generate an RGB image and a depth image by volume rendering;

[0054] Used to add the depth uncertainty calculated based on the inverse depth edge covariance to construct a loss function based on each of the key frame images and their corresponding depth images, camera poses, rendered RGB images, and rendered depth images;

[0055] And for minimizing the loss function, determining the optimal parameters of each layer of the multi-layer perceptron to render a real-time 3D map.

[0056] Preferably, the chassis of the medical mobile robot includes a retractable universal wheel assembly, and the real-time positioning and map building system further includes:

[0057] A control module, configured to determine the road condition ahead of the medical mobile robot at the current position based on the real-time 3D map, and estimate the distance between the medical mobile robot and surrounding objects;

[0058] And it is used to control the medical mobile robot to adaptively extend the universal wheel assembly to lower the center of gravity of the medical mobile robot when the vertical direction difference of the local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold; otherwise, keep the universal wheel assembly in a retracted state.

[0059] A storage medium, characterized in that it stores a computer program for real-time positioning and map construction applied to a medical mobile robot, wherein the computer program enables a computer to execute the real-time positioning and map construction method as described above.

[0060] An electronic device, comprising:

[0061] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a program for executing the instant positioning and mapping method as described above.

[0062] (III) Beneficial effects

[0063] The present invention provides a method, system, storage medium and electronic device for real-time positioning and map construction for medical mobile robots. Compared with the prior art, it has the following beneficial effects:

[0064] The present invention can be applied to but not limited to medical mobile robots for indoor activities. First, a monocular RGB camera that is suitable for complex environments and has lower cost is selected to collect video streams of real-world scenes, and the optical flow between different key frame images is used through a convolutional gated recurrent unit to estimate the depth information and camera pose corresponding to each key frame image. Next, the volume rendering capability of the neural radiation field is integrated with the ability of the hierarchical voxel grid to capture global and local scene information, and a hash encoding method with small storage space and easy search for grids corresponding to sampling points is used to achieve rapid rendering of larger scenes with limited resources. In addition, adding deep supervision through uncertainty estimation provides correct information for the neural radiation field that constructs the scene on the move, further accelerating the convergence speed, so as to achieve high-precision and real-time 3D map rendering of key frame images. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A block diagram of a method for real-time positioning and mapping applied to a medical mobile robot provided by an embodiment of the present invention;

[0067] Figure 2 A block diagram of another instant positioning and map building method for a medical mobile robot provided by an embodiment of the present invention;

[0068] Figure 3 A technical roadmap for a method for real-time positioning and mapping of a medical mobile robot provided by an embodiment of the present invention;

[0069] Figure 4 An example diagram of a medical mobile robot provided by an embodiment of the present invention (the universal wheel assembly is in a retracted state);

[0070] Figure 5 An exemplary diagram of a medical mobile robot provided by an embodiment of the present invention (the universal wheel assembly is in an extended state);

[0071] Figure 6 A schematic diagram of a medical mobile robot with an adaptively extendable universal wheel assembly descending a step provided by an embodiment of the present invention;

[0072] Figure 7 A schematic diagram of a medical mobile robot with an adaptively extendable universal wheel assembly climbing stairs provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] The embodiments of the present application solve the technical problem that real-time performance needs to be improved by providing a method, system, storage medium and electronic device for instant positioning and map construction applied to a medical mobile robot.

[0075] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0076] The embodiment of the present invention firstly adopts the most convenient and low-energy monocular RGB camera that is more suitable for complex environments, and estimates the camera pose and depth map by using the optical flow between multiple images through a convolutional gated recurrent unit; at the same time, hash coding is used to map the high-dimensional space or feature vector into a low-dimensional hash table, which can significantly reduce the computer storage requirements, and combines the advantages of neural radiance fields (NeRF) and hierarchical voxel grids by reducing the number of sampling times. Accurate radiation fields are constructed from image streams, and a new loss function is designed that considers image color loss, camera pose loss, color consistency loss between adjacent pixels, and optical flow loss between frame pairs. Back propagation minimizes the error between the radiation field rendering image and the true value image taken by the camera, and outputs a near-perfect pose estimation and a dense depth map. At the same time, deep supervision is added through uncertainty estimation to provide correct information for the neural radiation field of the scene constructed on the move to accelerate the convergence speed, and render the image into a high-precision and real-time 3D map.

[0077] Furthermore, the chassis of the medical mobile robot is set to include a retractable universal wheel assembly. Through the final rendered real-time 3D map, the estimated distance (i.e., depth) between the surrounding objects and the medical mobile robot is used to assist the medical mobile robot in determining the ground conditions in front of itself in the current position. If there is a preset slope in front and the preset distance is reached, the medical mobile robot automatically extends the universal wheel assembly, reaches the flat ground, and then retracts the universal wheel assembly to ensure that the center of gravity of the entire robot is stable and beautiful.

[0078] In summary, the method provided by the embodiment of the present invention can not only improve the positioning and mapping accuracy, but also enable the medical mobile robot to adopt corresponding strategies according to different terrain conditions, thereby improving the safety of the medical mobile robot and its adaptability to complex terrain.

[0079] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0080] Embodiment 1:

[0081] like Figure 1 As shown, a method for real-time positioning and map construction applied to a medical mobile robot includes:

[0082] S1. Using the built-in monocular RGB camera of the medical mobile robot, a video stream of a real-world scene is collected, and a number of key frame images are extracted from the video stream using COLMAP to construct a key frame list;

[0083] S2, traversing the key frame images in the key frame list, and using the optical flow between different key frame images through a convolutional gated recurrent unit to estimate the depth image, camera pose and inverse depth edge covariance corresponding to each key frame image;

[0084] S3, based on the depth image, camera pose and color information of each of the key frame images, pre-constructing a rough 3D map of the real world scene, and dividing the rough 3D map into voxel grids according to three different resolutions to obtain hierarchical coarse-grained, medium-grained and fine-grained scene graphs; wherein each of the voxel grids uses hash coding to store eight vertex features of the grid;

[0085] S4, based on the camera pose corresponding to the current key frame image and each pixel on the image, sampling is performed using a dilation mask centered on the interest point, and the voxel grid where each sampling point is located on the scene graph of different granularities is searched through hash coding;

[0086] S5. Based on the multi-layer perceptron corresponding to each layer of the voxel grid, obtain the occupancy value and color of each sampling point, and generate an RGB image and a depth image by volume rendering;

[0087] S6. Based on each of the key frame images and their corresponding depth images, camera poses, rendered RGB images, and rendered depth images, add the depth uncertainty calculated based on the inverse depth edge covariance to construct a loss function;

[0088] S7. Minimize the loss function and determine the optimal parameters of each layer of the multilayer perceptron to render a real-time 3D map.

[0089] The embodiments of the present invention can be applied to but not limited to medical mobile robots for indoor activities. A monocular RGB camera is selected for real-time positioning and mapping, so that key frame images can be rendered into high-precision and real-time 3D maps under limited resources.

[0090] In an optional embodiment, the chassis of the medical mobile robot includes a retractable universal wheel assembly, such as Figure 2 As shown, the instant positioning and map building method also includes:

[0091] S8. Based on the real-time 3D map, determine the road condition ahead of the medical mobile robot at the current position, and estimate the distance between the medical mobile robot and surrounding objects;

[0092] When the vertical direction difference of the local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, the medical mobile robot is controlled to adaptively extend the universal wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, the universal wheel assembly is kept retracted.

[0093] In the embodiment of the present invention, the medical mobile robot can be controlled to adopt corresponding strategies according to different terrain conditions, thereby improving the safety of the medical mobile robot and the adaptability to complex terrains.

[0094] like Figure 3 As shown, Figure 3 The technical roadmap of the instant positioning and map construction method provided by the embodiment of the present invention is given. Figure 3 The following are the steps of the above solution:

[0095] In step S1, a video stream of a real-world scene is collected through a monocular RGB camera built into the medical mobile robot, and a number of key frame images are extracted from the video stream using COLMAP to construct a key frame list.

[0096] It should be noted that many current robot 3D reconstruction solutions are based on RGB-D or LiDAR sensors, but a more convenient solution is available if only monocular RGB images are used for scene reconstruction. RGB-D cameras are prone to malfunction in poor lighting conditions, LiDAR is heavier and more expensive than monocular cameras, and encounters materials such as glass, which can cause scattered reflections of light and easily lead to inaccurate positioning estimates. Long-term use on mobile devices may cause position deviation.

[0097] In addition, stereo cameras simplify the depth estimation problem to a one-dimensional disparity search, but they are highly dependent on the calibration accuracy of the camera, which is prone to calibration errors in actual operation. In contrast, monocular RGB cameras are cheap, lightweight, and have low energy consumption, which can extend the battery life of the device. They also represent the simplest sensor configuration for calibration, making data processing easier.

[0098] Based on this, after physically investigating the terrain in the real application scenario, the embodiment of the present invention builds a monocular RGB camera into the medical mobile robot to collect the video stream of the real-world scene, and uses the open source dynamic structure package tool COLMAP to extract several key frame images from the video stream to construct a key frame list.

[0099] After obtaining the key frame list, the embodiment of the present invention then performs a positioning operation, for details, see the content of step S2.

[0100] In step S2, the key frame images in the key frame list are traversed, and the depth image, camera pose and inverse depth edge covariance corresponding to each key frame image are estimated by using the optical flow between different key frame images through a convolutional gated recurrent unit.

[0101] The embodiment of the present invention overcomes the drawback that a monocular RGB camera cannot estimate the surrounding depth map by designing a convolutional gated recurrent unit (ConvGRU) between consecutive key frames. The convolutional gated recurrent unit (ConvGRU) is composed of a gated recurrent unit (GRU) and a convolutional neural network (CNN), and specifically replaces the fully connected layer in the GRU with a convolutional layer, thereby being able to capture the spatial transformation in the spatiotemporal sequence data.

[0102] Correspondingly, the depth image, camera pose and inverse depth edge covariance corresponding to the current key frame image are estimated by using the optical flow between different key frame images through the convolution gated recurrent unit; specifically including:

[0103] The current key frame image is passed through a convolution layer to obtain a first feature map, the first feature map and the hidden state vector of the previous key frame image are spliced ​​together, input into a gated recurrent unit, and based on the calculation result of the gated recurrent unit, the hidden state vector of the current key frame image is obtained, and then passed through a convolution layer to obtain a second feature map.

[0104] Based on the first feature map, calculate the current optical flow measurement value p ij ; And introduce the smoothness assumption, based on the second feature map, update the current optical flow measurement value p by minimizing the energy function ij and its weight ∑p ij .

[0105] Based on the current optical flow measurement and its weights, the inter-image reprojection error equations are linearized into a block sparse Hessian matrix H∈R of camera / depth arrows (c+p)×(c+p) , where c and p are the dimensions of the camera and point, and the linearized equations are Hx = b, which can be expressed as:

[0106]

[0107] Among them, the above formula is the specific parameters of the equation system paradigm, C represents the block camera matrix (covering the camera intrinsic parameters and pixel coordinates), E represents the identity matrix, the superscript T represents the transpose, A represents the diagonal matrix of the inverse depth, Δɑ represents the increment on the Lie algebra of the camera pose, ΔD -1 represents the inverse depth increment of each pixel, and b represents the constant term composed of the depth residual m and the pose residual m.

[0108] To solve the linear least squares problem, the simplified camera matrix H is calculated using the Schur complement of the block sparse Hessian matrix T , for the simplified camera moment H T =LL T Perform Cholesky decomposition, and solve the camera pose T corresponding to the current key frame image through the previous iteration and the next iteration. Based on the solved camera pose T, the depth image D is further solved.

[0109] In order to make good use of the output (T, D) of the above dense mapping, the embodiment of the present invention further calculates the inverse depth D -1 Edge covariance and camera attitude T edge covariance quantify the uncertainty of depth estimation, camera position and orientation estimation for each pixel, thereby adding deep supervision, a useful information for training, to further optimize or fuse sensor data and accelerate convergence as information is provided. The formula for edge covariance is as follows:

[0110] ∑ D-1 =A -1 +A -T E T ∑ T EA -1

[0111] ∑ T =(LL T ) -1

[0112] Wherein, L represents a triangular matrix.

[0113] After completing the positioning operation, the embodiment of the present invention then performs a mapping operation, for details, see the contents of steps S3 to S7.

[0114] In step S3, based on the depth image, camera pose and color information of each of the key frame images, a rough 3D map of the real-world scene is pre-constructed, and the rough 3D map is divided into voxel grids according to three different resolution sizes to obtain hierarchical coarse-grained, medium-grained and fine-grained scene graphs; wherein each of the voxel grids uses hash coding to store the eight vertex features of the grid.

[0115] Exemplarily, in this step, the rough 3D map is divided into grids according to three different resolution sizes. The coarse-grained resolution grid learns the information of the overall scene. The grid size is set to a cube with a side length of 2m to effectively simulate the coarse scene geometry (such as walls and floors). The medium-grained and fine-grained resolution grids learn fine details. The grid sizes are set to side lengths of 16cm and 8cm, respectively, to obtain the high-frequency geometric details of the scene. Each grid stores the eight vertex features of the grid based on hash coding.

[0116] In step S4, based on the camera pose corresponding to the current key frame image and each pixel on its image, sampling is performed using a dilation mask centered on the point of interest, and the voxel grid where each sampling point is located on the scene graph of different granularities is searched through hash coding.

[0117] After obtaining the camera pose, multiple rays can be formed based on the camera center and each pixel of any key frame image. The sampling is performed using a dilation mask centered on the point of interest to search the voxel grid where the sampling point is located through hash coding, and the feature vector of the sampling point in the voxel grid is calculated using trilinear interpolation using the eight vertex features of the grid.

[0118] It should be noted that the method of expanding the mask based on the interest point as the center specifically refers to:

[0119] Determine the color and brightness distribution of the current key frame image, find the area with the most prominent color or the most significant brightness change in the image, take the center point of the area as the point of interest, and set the interval range to a square with a fixed number of pixels (for example, 5 pixels) as the area of ​​interest for iterative sampling.

[0120] It can be understood that in the embodiment of the present invention, the neural radiation field simulates light propagation in an implicit way to render and reconstruct the scene, and captures scene information of different dimensions in layers with grids of different resolutions, adds uncertainty estimation as supervision, and uses interest region sampling, which is a strategy to relax interest point sampling, and samples from the expansion mask centered on the interest point. After obtaining the interest point, iterates multiple times to expand the sampling area by 5×5. Both methods shorten the training time, allow the network to converge faster, and improve the real-time performance of simultaneous mapping and positioning.

[0121] In step S5, based on the multi-layer perceptron corresponding to each layer of the voxel grid, the occupancy value and color of each sampling point are obtained, and the RGB image and the depth image are generated by volume rendering.

[0122] In the embodiment of the present invention, each voxel grid is trained using a multi-layer perceptron. Accordingly, this step specifically includes:

[0123]

[0124] Among them, O P represents the occupancy value of point P; l represents the layer index of the voxel grid; represents the occupancy value of point P in the voxel grid of the lth layer; P represents the three-dimensional coordinates of the sampling point; represents the feature vector of point P in the l-layer voxel grid, which is calculated by the three-line difference of the eight vertex features of the grid; θ represents the parameters of the multi-layer perceptron; C P represents the color value of point P, which is decoded only by the multi-layer perceptron corresponding to the fine-grained resolution voxel grid; g β Represented as a multilayer perceptron for decoding color.

[0125] The above is the forward propagation process. The embodiment of the present invention also needs to design a loss function for back propagation to alternately update the pose and depth, use the rendered image, the color image of the entity image captured by the camera, and the estimated depth pose image as the loss, and construct and minimize the loss function. For details, refer to the contents of steps S6 to S7.

[0126] In step S6, based on each of the key frame images and their corresponding depth images, camera poses, rendered RGB images, and rendered depth images, the depth uncertainty calculated based on the inverse depth edge covariance is added to construct a loss function; specifically, the loss function includes:

[0127] (1) Constructing color loss:

[0128] L rgb (T,θ)=||II′(T,θ)|| 2

[0129] Where I′ represents the pixel color value of the rendered RGB image; I represents the pixel color value of the current key frame image; T represents the camera pose; θ is the multi-layer perceptron parameter; ||·|| 2 represents the 2-norm;

[0130] (2) Constructing depth loss:

[0131]

[0132] Where D′ represents the rendered depth image; ∑ D Represents deep uncertainty, used for supervised network learning;

[0133] (3) Constructing color consistency loss:

[0134]

[0135] Among them, for the pixel in the current key frame image m, it is recorded as r m; The pixel projected by the current key frame image m in the adjacent key frame image n is denoted as r m→n ; R represents the pixels sampled in the region of interest at each iteration; k m represents the key frame list, excluding the current key frame image m itself; I(·) represents the pixel color value; ||·||1 represents the 1 norm;

[0136] (4) Constructing optical flow loss

[0137]

[0138] Here, GM(·) represents the optical flow estimation.

[0139] (5) Constructing the total loss function

[0140] L=α rgb L rgb (T,θ)+α D L D (T,θ)+α warp L warp +α flow L flow

[0141] Among them, α rgb Represents the weight of color loss, α D represents the weight of the depth loss, α warp represents the weight of color consistency loss, α flow Represents the weight of the optical flow loss.

[0142] In step S7, the loss function is minimized to determine the optimal parameters of each layer of the multilayer perceptron to render a real-time 3D map.

[0143] Thus, the embodiment of the present invention realizes rendering a high-precision and real-time 3D map from key frame images under limited resources. Further, based on the real-time 3D map, the embodiment of the present invention also provides a solution for navigation and motion control of the medical mobile robot, see step S8 for details.

[0144] In step S8, based on the real-time 3D map, the road condition ahead of the medical mobile robot in the current position is determined, and the distance between the medical mobile robot and surrounding objects is estimated; when the vertical direction difference of the local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, the medical mobile robot is controlled to adaptively extend the universal wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, the universal wheel assembly is kept retracted.

[0145] For example, Figures 4-5As shown, the medical mobile robot provided by the embodiment of the present invention has a humanoid appearance, and the chassis of the medical mobile robot includes a retractable universal wheel assembly for coping with sloped jumping terrains such as going down stairs.

[0146] Specifically, when the vertical direction difference of the local terrain is greater than or equal to a preset first threshold value of 0-4 cm and the distance is less than or equal to a second threshold value of 30 cm, the medical mobile robot is controlled to adaptively extend the universal wheel assembly to lower the center of gravity of the medical mobile robot, such as Figures 6-7 Otherwise, the universal wheel assembly is kept in a retracted state. Thus, the safety of the medical mobile robot and its adaptability to complex terrain are improved.

[0147] The following is a specific application example of the embodiment of the present invention:

[0148] Data used: Maps were constructed in the intensive care unit of Hospital A, the infectious disease department of Hospital B, and the intensive care unit of Hospital C (with some slopes and thresholds for the scenarios themselves).

[0149] Specific settings: select K = 16 frames, of which 5 frames are randomly selected from the keyframe list, 10 frames are randomly selected from the most recent 20 keyframes, and the current frame. The batch size is set to 8, the region of interest is sampled as 5*5 pixels, and the final L = L rgb (T,θ)+0.1L D (T,θ)+0.5L warp +0.001L flow , the robot is set to adaptively extend the universal wheel assembly when it detects a vertical difference of 0-4cm in the local terrain when the distance is close to 30cm, otherwise the universal wheel assembly remains retracted.

[0150] Embodiment 2:

[0151] The embodiment of the present invention provides a real-time positioning and map building system for a medical mobile robot, comprising:

[0152] An extraction module, used to collect a video stream of a real-world scene through a monocular RGB camera built into the medical mobile robot, and extract a number of key frame images from the video stream using COLMAP to construct a key frame list;

[0153] A positioning module, used to traverse the key frame images in the key frame list, and estimate the depth image, camera pose and inverse depth edge covariance corresponding to each key frame image by using the optical flow between different key frame images through a convolutional gated recurrent unit;

[0154] A mapping module, for pre-building a rough 3D map of the real world scene based on the depth image, camera pose and color information of each of the key frame images, and dividing the rough 3D map into voxel grids according to three different resolutions to obtain hierarchical coarse-grained, medium-grained and fine-grained scene graphs; wherein each of the voxel grids uses hash coding to store eight vertex features of the grid;

[0155] It is used to sample the camera pose corresponding to the current key frame image and each pixel on its image by using a dilation mask centered on the point of interest, and search the voxel grid where each sampling point is located on the scene graph of different granularities through hash coding;

[0156] Used to obtain the occupancy value and color of each sampling point based on the multi-layer perceptron corresponding to each layer of the voxel grid, and generate an RGB image and a depth image by volume rendering;

[0157] Used to add the depth uncertainty calculated based on the inverse depth edge covariance to construct a loss function based on each of the key frame images and their corresponding depth images, camera poses, rendered RGB images, and rendered depth images;

[0158] And for minimizing the loss function, determining the optimal parameters of each layer of the multi-layer perceptron to render a real-time 3D map.

[0159] In an optional embodiment, the chassis of the medical mobile robot includes a retractable universal wheel assembly, and the real-time positioning and mapping system further includes:

[0160] A control module, configured to determine the road condition ahead of the medical mobile robot at the current position based on the real-time 3D map, and estimate the distance between the medical mobile robot and surrounding objects;

[0161] And it is used to control the medical mobile robot to adaptively extend the universal wheel assembly to lower the center of gravity of the medical mobile robot when the vertical direction difference of the local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold; otherwise, keep the universal wheel assembly in a retracted state.

[0162] Embodiment 3:

[0163] An embodiment of the present invention provides a storage medium storing a computer program for real-time positioning and map building applied to a medical mobile robot, wherein the computer program enables a computer to execute the real-time positioning and map building method as described in Example 1.

[0164] Embodiment 4:

[0165] An embodiment of the present invention provides an electronic device, including:

[0166] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the instant positioning and mapping method as described in Example 1.

[0167] It can be understood that the instant positioning and mapping system, storage medium and electronic device provided in the embodiments of the present invention for medical mobile robots correspond to the instant positioning and mapping method provided in the embodiments of the present invention for medical mobile robots. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the instant positioning and mapping method, which will not be repeated here.

[0168] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0169] 1. The embodiment of the present invention investigates the terrain in real application scenarios, considers the selection of the most suitable sensor, and overcomes the disadvantage that the monocular RGB camera cannot estimate the surrounding depth map by designing a convolutional gated recurrent unit between consecutive key frames.

[0170] 2. The embodiment of the present invention combines the volume rendering capability of the neural radiation field with the ability of the hierarchical voxel grid to capture global and local scene information, and uses a hash coding method that uses less storage space and more quickly finds the grid corresponding to the sampling point, so that larger scenes can be rendered with faster convergence speed under limited resources to improve real-time performance.

[0171] 3. In order to speed up the training, the embodiment of the present invention first uses the marginal covariance of the output of the convolutional gated recurrent unit as uncertainty estimation, adds this information into the input of the multi-layer perceptron, converges faster, and improves the sampling method, which is not based on the random adoption of the original neural radiation field, but on the sampling of the interest area based on the interest point, and samples in the dilated mask centered on the interest point.

[0172] 4. The embodiment of the present invention sets a new loss function, taking into account the smoothness between images and pixels, so that the rendered image has a lower error rate, smaller geometric errors and scene object color errors, and more accurate depth estimation, so that the universal wheel assembly can be extended or retracted at a more accurate time.

[0173] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0174] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time positioning and mapping for a medical mobile robot, characterized in that: include: The medical mobile robot is equipped with a built-in monocular RGB camera to collect a video stream of a real-world scene, and a plurality of key frame images are extracted from the video stream using COLMAP to construct a key frame list; Traversing the key frame images in the key frame list, and using the optical flow between different key frame images through a convolutional gated recurrent unit to estimate the depth image, camera pose, and inverse depth edge covariance corresponding to each key frame image; Based on the depth image, camera pose and color information of each of the key frame images, a rough 3D map of the real world scene is pre-constructed, and the rough 3D map is divided into voxel grids according to three different resolutions to obtain hierarchical coarse-grained, medium-grained and fine-grained scene graphs; wherein each of the voxel grids uses hash coding to store eight vertex features of the grid; Based on the camera pose corresponding to the current keyframe image and each pixel on its image, sampling is performed using a dilation mask centered on the interest point, and the voxel grid where each sampling point is located on the scene graph of different granularities is searched through hash coding; Based on the multi-layer perceptron corresponding to each layer of the voxel grid, the occupancy value and color of each sampling point are obtained, and the RGB image and the depth image are generated by volume rendering; Based on each of the key frame images and their corresponding depth images, camera poses, rendered RGB images, and rendered depth images, adding the depth uncertainty calculated based on the inverse depth edge covariance, and constructing a loss function; The loss function is minimized to determine the optimal parameters of each layer of the multilayer perceptron to render a real-time 3D map.

2. The instant positioning and map building method according to claim 1, characterized in that: The chassis of the medical mobile robot includes a retractable universal wheel assembly, and the instant positioning and map building method further includes: Based on the real-time 3D map, determine the road condition ahead of the medical mobile robot at the current position, and estimate the distance between the medical mobile robot and surrounding objects; When the vertical direction difference of the local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold, the medical mobile robot is controlled to adaptively extend the universal wheel assembly to lower the center of gravity of the medical mobile robot; otherwise, the universal wheel assembly is kept retracted.

3. The instant positioning and map building method according to claim 1, characterized in that: Using the optical flow between different key frame images through a convolution gated recurrent unit, the depth image, camera pose and inverse depth edge covariance corresponding to the current key frame image are estimated; including: The current key frame image is passed through a convolution layer to obtain a first feature map, the first feature map and the hidden state vector of the previous key frame image are concatenated together, input into a gated recurrent unit, and according to the calculation result of the gated recurrent unit, the hidden state vector of the current key frame image is obtained, and then passed through a convolution layer to obtain a second feature map; Based on the first feature map, calculate the current optical flow measurement value; and introduce a smoothness assumption, based on the second feature map, update the current optical flow measurement value and its weight by minimizing the energy function; Based on the current optical flow measurement and its weight, the inter-image reprojection error equations are linearized into a block sparse Hessian matrix of camera / depth arrows; Using the block sparse Hessian matrix Schu r Compute the simplified camera matrix and perform Cholesk on the simplified camera matrix y Decompose and solve the camera pose and depth image corresponding to the current key frame image through the previous iteration and the next iteration; Based on the camera pose and the depth image, the inverse depth edge covariance and the camera pose edge covariance corresponding to the current key frame image are calculated respectively.

4. The instant positioning and map building method according to claim 1, characterized in that: The sampling method based on the expansion mask centered on the interest point refers to: Determine the color and brightness distribution of the current keyframe image, find the area with the most prominent color or the most significant brightness change in the image, use the center point of the area as the point of interest, and set the interval range to a square with a fixed number of pixels as the area of ​​interest for iterative sampling.

5. The instant positioning and map building method according to claim 4, characterized in that: The multi-layer perceptron corresponding to each layer of the voxel grid is used to obtain the occupancy value and color of each sampling point; it is expressed as: Among them, O P represents the occupancy value of point P; l represents the layer index of the voxel grid; represents the occupancy value of point P in the voxel grid of the lth layer; P represents the three-dimensional coordinates of the sampling point; represents the feature vector of point P in the l-layer voxel grid, which is calculated by the three-line difference of the eight vertex features of the grid; θ represents the parameters of the multi-layer perceptron; C P represents the color value of point P, which is decoded only by the multi-layer perceptron corresponding to the fine-grained resolution voxel grid; g β Represented as a multilayer perceptron for decoding color.

6. The instant positioning and map building method according to any one of claims 1 to 5, characterized in that: The process of constructing the loss function specifically includes: (1) Constructing color loss: L rgb (T,θ)=||I-I(T,θ)|| 2 Where I represents the pixel color value of the rendered RGB image; I represents the pixel color value of the current key frame image; T represents the camera pose; θ is the multi-layer perceptron parameter; ||·|| 2 represents the 2-norm; (2) Constructing depth loss: Where D represents the rendered depth image; ∑ D Represents deep uncertainty, used for supervised network learning; (3) Constructing color consistency loss: Among them, for the pixel in the current key frame image m, it is recorded as r m ; The pixel projected by the current key frame image m in the adjacent key frame image n is denoted as r m→n ; R represents the pixels sampled in the region of interest at each iteration; k m represents the key frame list, excluding the current key frame image m itself; I(·) represents the pixel color value; ||·||1 represents the 1 norm; (4) Constructing optical flow loss Among them, GM(·) represents the optical flow estimation; (5) Constructing the total loss function L=a rgb L rgb (T,θ)+α D L D (T,θ)+α warp L warp +a flow L flow Among them, α rgb Represents the weight of color loss, α D represents the weight of the depth loss, α warp represents the weight of color consistency loss, α flow Represents the weight of the optical flow loss.

7. A real-time positioning and mapping system for a medical mobile robot, characterized in that: include: An extraction module, used to collect a video stream of a real-world scene through a monocular RGB camera built into the medical mobile robot, and extract a number of key frame images from the video stream using COLMAP to construct a key frame list; A positioning module, used to traverse the key frame images in the key frame list, and estimate the depth image, camera pose and inverse depth edge covariance corresponding to each key frame image by using the optical flow between different key frame images through a convolutional gated recurrent unit; A mapping module, for pre-building a rough 3D map of the real world scene based on the depth image, camera pose and color information of each of the key frame images, and dividing the rough 3D map into voxel grids according to three different resolutions to obtain hierarchical coarse-grained, medium-grained and fine-grained scene graphs; wherein each of the voxel grids uses hash coding to store eight vertex features of the grid; It is used to sample the camera pose corresponding to the current key frame image and each pixel on its image by using a dilation mask centered on the point of interest, and search the voxel grid where each sampling point is located on the scene graph of different granularities through hash coding; Used to obtain the occupancy value and color of each sampling point based on the multi-layer perceptron corresponding to each layer of the voxel grid, and generate an RGB image and a depth image by volume rendering; Used to add the depth uncertainty calculated based on the inverse depth edge covariance to construct a loss function based on each of the key frame images and their corresponding depth images, camera poses, rendered RGB images, and rendered depth images; And for minimizing the loss function, determining the optimal parameters of each layer of the multi-layer perceptron to render a real-time 3D map.

8. The real-time positioning and map building system according to claim 7, characterized in that: The chassis of the medical mobile robot includes a retractable universal wheel assembly, and the real-time positioning and map building system also includes: A control module, configured to determine the road condition ahead of the medical mobile robot at the current position based on the real-time 3D map, and estimate the distance between the medical mobile robot and surrounding objects; And it is used to control the medical mobile robot to adaptively extend the universal wheel assembly to lower the center of gravity of the medical mobile robot when the vertical direction difference of the local terrain is greater than or equal to a preset first threshold and the distance is less than or equal to a second threshold; otherwise, keep the universal wheel assembly in a retracted state.

9. A storage medium, characterized in that: It stores a computer program for real-time positioning and map construction applied to a medical mobile robot, wherein the computer program enables the computer to execute the real-time positioning and map construction method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including programs for executing the instant positioning and map building method according to any one of claims 1 to 6.

Images