A robot global relocalization method based on deep neural network and three-dimensional point cloud

Abstract

The application discloses a robot global repositioning method based on a deep neural network and a three-dimensional point cloud, and comprises the following steps: 3D mapping of a target scene is performed by using a laser radar; 3D point cloud key frames are saved and projected on a spherical surface into multi-dimensional arrays; channel number conversion is performed on the obtained n multi-dimensional arrays; a neural network is built; the n multi-dimensional arrays are sent into the neural network for training; the processed key frame point cloud multi-dimensional arrays are sent into the trained neural network; n outputs of an encoder layer are flattened into n descriptors; the robot is placed in any position in a map after being restarted, and the descriptors of the key frames of the robot at the current time are obtained; similarity calculation is performed on the key frame point cloud descriptors photographed by the robot and the key frame point cloud descriptors in the 3D target map, and the descriptors are sorted in descending order; and repositioning is completed. The application can quickly realize repositioning, does not need prior information of a GPS, and enhances the robustness of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot positioning, and particularly to a robot global repositioning method based on a deep neural network and a three-dimensional point cloud. BACKGROUND

[0002] In the process of autonomous navigation of a mobile robot, autonomous positioning is often required, that is, the robot estimates its pose and attitude relative to a known map environment by relying on its own sensors, also known as robot pose estimation. When the robot is restarted or "kidnapped", the robot needs to reposition itself to estimate its pose in the known map, thereby achieving repositioning. When the mobile robot is indoors, there is no global positioning system (GPS) signal, so the mobile robot repositioning fails. When the mobile robot is outdoors, the GPS signal is very unstable due to dense places such as high-rise buildings and forests, thereby causing the robot repositioning to fail. SUMMARY

[0003] To overcome the shortcomings of the prior art, the present application provides a robot global repositioning method based on a deep neural network and a three-dimensional point cloud. The technical scheme of the present application is as follows:

[0004] A robot global repositioning method based on a deep neural network and a three-dimensional point cloud, comprising the steps of

[0005] S1: using a laser radar to perform 3D mapping on a target scene, and saving point clouds of n key frames;

[0006] S2: performing spherical projection on the saved 3D point cloud key frames to obtain multi-dimensional arrays;

[0007] S3: performing channel number conversion on the n multi-dimensional arrays obtained in step S2, so that the channel number is changed from (64x512x9) to (9x64x512);

[0008] S4: building a neural network, wherein the neural network comprises an Encoder layer and a Decoder layer;

[0009] S5: sending the n multi-dimensional arrays into the neural network for training, to obtain a trained neural network;

[0010] S6: sending the processed key frame point cloud multi-dimensional arrays into the trained neural network, and extracting the output of the Encoder layer;

[0011] S7: extracting the n output encoders of the Encoder layer, which are (1x8x8) descriptors, performing data processing, and flattening them into n (1x64) descriptors, denoted as X(x1, x2, x3,......., xn );

[0012] S8: Put the robot in any position in the map after rebooting, get the key frame of the robot at the current time, perform spherical projection on the key frame to obtain a multi-dimensional matrix (64x512x9), then convert the channel number to (9x64x512), and then send it into the neural network. The output of the Encoder layer is a descriptor of (1x8x8), which is then flattened into a descriptor of (1x64) and named x p ;

[0013] S9: Calculate the similarity between the key frame point cloud descriptor x p and the key frame point cloud descriptor X (x1, x2, x3,..., x n ) in the 3D target map, and sort them from large to small, and finally get a continuous number sequence set output;

[0014] S10: Get the number sequence list, set a threshold b, take out the key frame point cloud corresponding to the [c-b, c+b] list, and splice it into a local map, and then use the current time robot to shoot a frame of point cloud and the local map to solve the pose, complete the repositioning.

[0015] Further, the projection process in step S2 includes steps

[0016] S21: For a point P (x1, y1, z1) in the key frame point cloud, first calculate the pitch angle pitch and yaw angle yaw, and the calculation formula is: where R represents the distance formula

[0017] S22: Move the laser radar as the origin of the new coordinates to the upper left corner, perform coordinate origin transformation, and calculate the pitch angle pitch and yaw angle yaw: FOV_Up = FOV - abs(FOV_Down), a rotating scanning laser radar, its vertical field of view angle FOV is divided into FOV_Up and FOV_Down, FOV_Up and FOV_Down are the maximum field of view and the minimum field of view formed by the first and last lasers of the laser radar;

[0018] S23: Perform normalization and scaling:

[0019] nor_pitch = (FOV_Up - pitch) / (FOV_Up - FOV_Down)

[0020] nor_yaw = (yaw + π) / 2π

[0021] FOV = FOV_Up + abs(FOV_Down) = 2 x FOV_Up

[0022] The final 3D point cloud is represented in the final (u, v) coordinates as:

[0023] where row_scale is the image width and col_scale is the image height;

[0024] S24: Perform spherical projection on the points in the n key frame point clouds to obtain n multi-dimensional arrays with a shape of (64 x 512 x 9).

[0025] Further, step S6 comprises the steps of

[0026] S61: Input the multi-dimensional arrays of the n key frame point clouds with a shape of (9 x 64 x 512) into the network for training, and the output of the first layer Encoder layer is n descriptors with a shape of (1 x 8 x 8), which are named as encoder; then input the encoder into the second layer Decoder layer to restore them to n multi-dimensional arrays with a shape of (9 x 64 x 512), which are named as decoder.

[0027] S62: The loss function used by the neural network is the mean square loss function: output1 is the input multi-dimensional array, and output2 is the output of the Decoder layer.

[0028] Further, step S9 comprises the steps of

[0029] S91: The formula for calculating the similarity is: loss(x i ,y i ) = (x i -y i ) 2 , which returns the average value between the two vectors, where x i is the descriptor x p obtained after processing after the robot restarts, and y i is each descriptor in X (x1, x2, x3,...., x n ). The descriptor in each X needs to be calculated with x p .

[0030] S92: Store the labels (0, 1, 2,..., n) and the average values corresponding to the labels into a new list, and the labels correspond to the serial numbers of the key frame point clouds.

[0031] S93: sort the average values in the list from high to low, take out the first K and update the list, and store the serial key frame numbers with the greatest similarity to the key frame obtained by the current robot in the list.

[0032] Further, the step S10 comprises the steps of

[0033] S101: extract a list containing the key frame numbers and consecutive numbers;

[0034] S102: set a threshold b, take out the key frame point cloud corresponding to [c-b, c+b] in the custom label list, and splice it into a local map;

[0035] S103: the current frame and the local map are calculated by the ICP algorithm, the position and attitude of the robot in the map at the current time are obtained, and the repositioning is completed.

[0036] The beneficial effects of the present application are that, compared with the prior art, the present application can quickly realize repositioning when the indoor GPS satellite signal is unstable or the camera light is unstable or the texture information is not obvious, and does not require GPS prior information, and also enhances the robustness of the system. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flow chart of a robot global repositioning method based on a deep neural network and a three-dimensional point cloud according to the present application;

[0038] Figure 2 is an Encoder layer structure diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] Please refer to Figure 1 , a robot global repositioning method based on a deep neural network and a three-dimensional point cloud, comprising the steps of

[0041] S1: 3D mapping of the target scene is performed by using a laser radar, and the point clouds of n key frames are saved;

[0042] S2: the saved 3D point cloud key frames are projected onto a spherical surface to form a multi-dimensional array; the projection process in the step S2 comprises the steps of

[0043] S21: for a point P(x1, y1, z1) in the key frame point cloud, first calculate the pitch angle and yaw angle, the calculation formula is: wherein R represents the distance formula is

[0044] S22: move the laser radar as the origin of the new coordinate to the upper left corner, perform coordinate origin transformation, pitch angle and yaw angle: FOV_Up = FOV-abs(FOV_Down), a rotating scanning laser radar of an m line, the vertical field of view angle FOV is divided into FOV_Up, FOV_Down two parts, FOV_Up, FOV_Down are the maximum field of view and the minimum field of view formed by the first and the last laser of the laser radar;

[0045] S23: normalization and scaling:

[0046] nor_pitch = (FOV_Up-pitch) / (FOV_Up-FOV_Down)

[0047] nor_yaw = (yaw+π) / 2π

[0048] FOV = FOV_Up + abs(FOV_Down) = 2xFOV_Up

[0049] The final 3D point cloud is represented in the final (u, v) coordinates as:

[0050] Wherein row_scale is the image width, and col_scale is the image height;

[0051] S24: perform spherical projection on the points in the n key frame point clouds to obtain n multi-dimensional arrays in the shape of (64x512x9).

[0052] S3: change the channel number of the n multi-dimensional arrays obtained in step S2 from (64x512x9) to (9x64x512);

[0053] S4: build a neural network, the neural network comprising an Encoder layer and a Decoder layer;

[0054] S5: send the n multi-dimensional arrays into the neural network for training to obtain a trained neural network;

[0055] S6: send the processed key frame point cloud multi-dimensional array into the trained neural network and extract the output of the Encoder layer; comprising the steps

[0056] S61: The multi-dimensional array of n key frame point clouds with a shape of (9x64x512) is sent to the network for training. The output of the first layer of the Encoder layer is n descriptors with a shape of (1x8x8), which are named as encoder. The encoder is then sent to the second layer of the Decoder layer to restore it to n multi-dimensional arrays of (9x64x512), which are named as decoder;

[0057] S62: The loss function used by the neural network is the mean square loss function: output1 is the input multi-dimensional array, and output2 is the output of the Decoder layer.

[0058] S7: The n outputs of the Encoder layer, which are descriptors of the type (1x8x8), are extracted for data processing and flattened into n descriptors of the type (1x64), which are denoted as X(x1, x2, x3,......., x n );

[0059] S8: After the robot is restarted, it is placed in any position in the map, and the key frame of the robot at the current time is obtained. The key frame is projected onto a sphere to obtain a multi-dimensional matrix (64x512x9), which is then converted into (9x64x512) and sent to the neural network. The output of the Encoder layer is a descriptor of the type (1x8x8), which is then flattened into a descriptor of the type (1x64) and denoted as x p ;

[0060] S9: The key frame point cloud descriptor x p taken by the robot and the key frame point cloud descriptors X(x1, x2, x3,......., x n ) in the 3D target map are calculated for similarity and sorted from large to small, and finally a set of zero to multiple consecutive number sequences is outputted.

[0061] S91: The formula for similarity calculation is: loss(x i , y i ) = (x i - y i ) 2 , which returns the average value between the two vectors, where x i is the descriptor x p processed after the robot is restarted, and y i is each descriptor in X(x1, x2, x3,......., x n ). Each descriptor in X is calculated for similarity with x p ;

[0062] S92: Store the label (0, 1, 2,..., n) and the average value corresponding to the label into a new list, and the label corresponds to the serial number of the key frame point cloud;

[0063] S93: Sort the average values in the list from high to low, take out the first K and update the list, and store the serial numbers of the continuous key frames with the greatest similarity to the current robot-acquired key frame in the list.

[0064] For example, x p The output list is [c-2, c-1, c, c+1, c+2, c+3], and the key frame point cloud corresponding to label a may be the same scene as the key frame point cloud corresponding to label c, or the key frame point cloud corresponding to label a may be the same scene as the key frame point cloud corresponding to label c-1. The key frame point cloud corresponding to the label in the list is the one with the greatest similarity to the point cloud acquired by the robot at the current time.

[0065] S10: Obtain the serial number list, set a threshold value b, take out the key frame point cloud corresponding to the custom [c-b, c+b] list, splice it into a local map, and then use the point cloud acquired by the robot at the current time to solve the pose to complete the repositioning.

[0066] S101: Extract the list containing the key frame serial number and continuous serial number;

[0067] S102: Set a threshold value b, take out the key frame point cloud corresponding to [c-b, c+b] in the custom label list, and splice it into a local map;

[0068] S103: Use the ICP algorithm to solve the pose of the current frame acquired by the robot and the local map to obtain the position and attitude of the robot in the map at the current time, and complete the repositioning.

[0069] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A robot global relocalization method based on deep neural network and three-dimensional point cloud, characterized in that, The method comprises the steps of: S1: using a laser radar to perform 3D mapping on a target scene, and saving point clouds of n frames of key frames; S2: performing spherical projection on the saved 3D point cloud key frames into multi-dimensional arrays; S3: performing channel number conversion on the n multi-dimensional arrays obtained in step S2, so that the channel number is changed from (64*512*9) to (9*64*512); S4: building a neural network, wherein the neural network comprises an Encoder layer and a Decoder layer; S5: sending the n multi-dimensional arrays into the neural network for training, so as to obtain a trained neural network; S6: sending the processed multi-dimensional array of the key frame point clouds into the trained neural network, and extracting the output of the Encoder layer; S7: Extract the n output encoder descriptors of type (1×8×8) from the Encoder layer, process the data, and flatten them into n (1×64) descriptors, denoted as X(x1, x2, x3, ..., x...). n ); S8: After the robot is restarted, put it in any position in the map, get the key frame of the robot at the current time, perform spherical projection on the key frame to obtain a multi-dimensional matrix (64x512x9), then convert the channel number to become (9x64x512), and then send it into the neural network. The output of the Encoder layer is a descriptor of (1x8x8), which is then flattened into a descriptor of (1x64), named x p ; S9: Key frame point cloud descriptor x photographed by robot p and the key frame point cloud descriptor X (x1, x2, x3,......., x n ) in the 3D target map are calculated for similarity and sorted from large to small, and finally a zero to multiple continuous numbered sequence set is output. S10: obtaining a serial number list, setting a threshold b, taking out the key frame point clouds corresponding to the [c-b, c+b] list in the custom list, splicing the key frame point clouds into a local map, and then using a frame of point clouds captured by the robot at the current moment and the local map to perform pose calculation to obtain the position and posture of the robot in the map, so that the repositioning is completed.

2. The robot global relocalization method based on deep neural network and three- dimensional point cloud of claim 1, wherein, The projection process in step S2 comprises the steps of: S21: for a point P (x1, y1, z1) in the key frame point cloud, first find the pitch angle pitch and yaw angle yaw, and the calculation formula is: where R represents the distance formula is S22: The laser radar is moved to the upper left corner as the origin of the new coordinate, the coordinate origin is transformed, the pitch and yaw angles are adjusted FOV_Up = FOV - abs(FOV_Down), the vertical field of view angle FOV of the rotating scanning laser radar is divided into FOV_Up and FOV_Down, FOV_Up and FOV_Down are the maximum field of view and the minimum field of view formed by the first and last lasers of the laser radar; S23: performing normalization and scaling: nor_pitch = (FOV_Up-pitch) / (FOV_Up-FOV_Down) nor_yaw = (yaw+π) / 2π FOV = FOV_Up+abs(FOV_Down) = 2*FOV_Up The final 3D point cloud is represented in the final (u, v) coordinates as: where row_scale is the image width and col_scale is the image height; S24: performing spherical projection on the points in the n frames of key frame point clouds to obtain n multi-dimensional arrays in the shape of (64*512*9).

3. The robot global relocalization method based on deep neural network and three- dimensional point cloud of claim 2, wherein, Step S6 comprises the steps of: S61: sending the multi-dimensional arrays of the n key frame point clouds in the shape of (9*64*512) into the network for training, wherein the output of the first layer Encoder layer is n descriptors in the shape of (1*8*8), the n descriptors are named as encoder, and the encoder is sent into the second layer Decoder layer to restore the n multi-dimensional arrays in the shape of (9*64*512), and the multi-dimensional arrays are named as decoder; S62: The loss function used by the neural network is the mean square loss function: output1 is a multi-dimensional array of inputs, and output2 is the output of the Decoder layer.

4. The robot global relocalization method based on deep neural network and three- dimensional point cloud of claim 3, wherein, Step S9 comprises the steps of: S91: The formula for similarity calculation is: loss(x i ,y i ) = (x i -y i ) 2 , return the average value between two vectors, where x i is the descriptor x p processed after the robot restarts, y i is each descriptor in X(x1, x2, x3,...., x n ), and each descriptor in X is calculated with x p for similarity; S92: storing the labels (0, 1, 2,..., n) and the average values corresponding to the labels into a new list, wherein the labels correspond to the serial numbers of the key frame point clouds; S93: sorting the average values in the list from high to low, taking out the first K values, and updating the list, wherein the list stores the serial numbers of the continuous key frames that are most similar to the key frame obtained by the current robot.

5. The robot global relocalization method based on deep neural network and three- dimensional point cloud of claim 4, wherein, Step S10 comprises the steps of: S101: extracting the list containing the serial numbers of the key frames and the serial numbers in sequence; S102: setting a threshold b, taking out the key frame point clouds corresponding to [c-b, c+b] in the custom label list, and splicing the key frame point clouds into a local map; S103: using the ICP algorithm to perform pose calculation on the current frame captured by the robot and the local map, so as to obtain the position and posture of the robot in the map at the current moment, and complete the repositioning.

Images