Posture recovery method, device, readable storage medium and robot

Abstract

The present invention provides a posture recovery method, device, readable storage medium, and robot. The method comprises: when the robot loses positioning information, obtaining visual data collected by a visual sensor; inputting the visual data into a preset neural network model, for the preset neural network model to encode the visual data to obtain an encoding result; performing a similarity comparison between the encoding result and attribute information of network nodes in the preset neural network model to obtain a comparison result; determining a target network node based on the comparison result, wherein the attribute information of the target network node includes first-level posture data; and outputting the first-level posture data. By executing this recovery method, the probability of the robot's navigation system crashing due to a lack of posture data is reduced.

Description

Technical Field

[0001] The present invention relates to the field of map construction and restoration technology, and in particular to a posture restoration method, device, readable storage medium and robot. Background Art

[0002] During the robot mapping process, sensor data may be lost due to unexpected circumstances, or an initial position may be required when the map is known for navigation. Due to the robot kidnapping problem, the robot is often unable to efficiently recover its position data, which can easily cause the navigation system to crash. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0004] To this end, a first aspect of the present invention is to provide a posture recovery method.

[0005] A second aspect of the present invention is to provide a posture recovery device.

[0006] A third aspect of the present invention is to provide a posture recovery device.

[0007] A fourth aspect of the present invention provides a readable storage medium.

[0008] A fifth aspect of the present invention provides a robot.

[0009] In view of this, according to a first aspect of the present invention, the present invention provides a posture recovery method for a robot, the robot including a visual sensor, the posture recovery method including: when the robot loses positioning information, obtaining visual data collected by the visual sensor; inputting the visual data into a preset neural network model for the preset neural network model to encode the visual data to obtain an encoding result, and performing a similarity comparison between the encoding result and the attribute information of the network node in the preset neural network model to obtain a comparison result, and determining the target network node based on the comparison result, wherein the attribute information of the target network node includes the first posture data; and outputting the first posture data.

[0010] The technical solution of the present application proposes a posture recovery method. By running this recovery method, when the sensor data is lost due to an emergency during the robot mapping process or when the initial position is required under the condition of known map navigation positioning, the robot's posture data can be quickly determined based on the acquired visual data. This overcomes the existence of the robot kidnapping problem and reduces the probability of the robot's navigation system crashing due to the lack of posture data.

[0011] By analyzing a large amount of robot relocalization data and algorithms, it was found that sensor observation data in the environment (such as images, point clouds and other visual information) can form nonlinear constraints with the trajectory and pose graph generated when the map is established.

[0012] Based on this, the technical solution of the present application pre-trains the neural network model in order to obtain a trained preset neural network model, so that when positioning information is lost, the acquired visual data is sent to the preset neural network model, and the visual data is encoded and searched based on the trained preset neural network model, so as to efficiently regress the current posture of the robot.

[0013] Among them, since the encoding process of visual data occurs during the processing of the preset neural network model, there is no need to set up an additional encoder-decoder. The robot can directly send the visual data to the preset neural network model, which simplifies the data processing process.

[0014] In addition, by performing similarity comparison, the target network nodes obtained are matched with the visual data, ensuring the accuracy of the first pose data and the visual data.

[0015] In addition, the posture recovery method proposed in this application also has the following additional technical features.

[0016] In the above technical solution, the posture recovery method also includes: constructing a graph neural network; obtaining historical visual data; determining the key frame data in the training data and the second posture data corresponding to the key frame based on the historical visual data; training the graph neural network based on the second posture data and the key frame data to obtain a preset neural network model.

[0017] In this technical solution, a training process for a preset neural network model is specifically provided. In this technical solution, since historical visual data all have corresponding pose data, the historical visual data is obtained so that training data can be determined based on the historical visual data. The training data includes corresponding second pose data and keyframe data. It can be understood that keyframe data, also known as a keyframe, can be understood as the skeleton of SLAM, which is a frame selected from a series of local ordinary frames as a representative local frame.

[0018] In one of the technical solutions, the second posture data can be understood as the posture data of the robot under the key frame.

[0019] In this technical solution, by selecting key frames and second pose data for model training, the time spent on training can be reduced and the training speed can be improved.

[0020] In this technical solution, graph neural network can be understood as a deep learning method based on graph structure, which mainly includes two parts, namely "graph" and "neural network", among which "graph" is the graph data structure in graph theory, and "neural network" is the deep learning NN structure.

[0021] In the above technical solution, historical visual data is processed based on the VINS-RGBD visual odometry calculation method to obtain key frame data and second pose data corresponding to each other.

[0022] In any of the above technical solutions, the key frame data includes target key frame data and remaining key frame data, and the graph neural network is trained according to the second pose data and the key frame data to obtain a preset neural network model, including: according to the second pose data, projecting each pixel point in the target key frame data to the remaining key frame data to obtain a similarity matrix; training the graph neural network according to the similarity matrix to obtain a preset neural network model.

[0023] In this technical solution, since the similarity matrix reflects the data relationship between different key frames, in the process of training the graph neural network using the similarity matrix, the preset neural network model obtained by training is no longer the training result between a single key frame, but can reflect the relationship between key frames, thereby realizing the fusion of attributes between network nodes and ensuring the accuracy of the first pose data output by the trained preset neural network model.

[0024] In the above technical solution, the target key frame data may be any key frame in the key frame data, and the remaining key frame data is the key frame data other than the target key frame data in the key frame data.

[0025] In any of the above technical solutions, each pixel point in the target key frame data is projected onto the remaining key frame data according to the second posture data to obtain a similarity matrix, which specifically includes: determining the coordinate value of each pixel point in the target key frame data projected on the remaining key frame data; determining the number of pixels of the target key frame data projected on the remaining key frame data according to the coordinate value; determining the intersection-union ratio of the target key frame data and the remaining key frame data according to the number of pixels; and determining the similarity matrix according to the intersection-union ratio.

[0026] In this technical solution, a specific determination scheme for the similarity matrix is ​​specifically given. In this scheme, the intersection-and-union ratio is calculated by counting the number of pixel points projected by the target key frame data on the remaining key frame data, and the similarity matrix is ​​determined using the intersection-and-union ratio.

[0027] In this process, in order to determine whether the pixel points in the target key frame data have projections on the remaining key frame data, the technical solution of the present application counts the coordinate values ​​of each pixel point in the target key frame data after projection. If the coordinate value is within the coordinate interval corresponding to the remaining key frame data, the pixel point is considered to be a valid pixel point, and if the coordinate value is outside the coordinate interval corresponding to the remaining key frame data, the pixel point is considered to be an invalid pixel point. By counting the number of valid pixel points as the number of pixels, in this solution, the number of pixels can be understood as the number of pixels overlapping between the target key frame data and the remaining key frame data.

[0028] In this technical solution, the sum of the number of pixels of the target key frame data and the number of pixels of the remaining key frame data is also determined, and further, the difference between the sum and the number of pixels is determined, and the ratio of the number of pixels to the difference is used as the intersection-union ratio.

[0029] The similarity matrix can be obtained by selecting all target key frames in the key frame data.

[0030] In one possible design, each frame of the image is projected by the pose to obtain a similarity matrix. The specific projection method is as follows:

[0031] p k+1 '=Tp k

[0032] Among them, T is the posture matrix, P k is the target key frame data, p k+1 ' is the remaining key frame data.

[0033] In any of the above technical solutions, a graph neural network is constructed, including: taking the pose nodes corresponding to the key frame data as network nodes; taking the second pose data and the encoded visual data as attribute information of the network nodes; and connecting all the nodes to each other to obtain a graph neural network.

[0034] In this technical solution, a specific method for constructing a graph neural network is given. In this method, the network node selects the pose node corresponding to the key frame data, and uses the encoded visual data and the second pose data as attribute information, and connects all nodes to achieve full connection.

[0035] In the above process, due to the introduction of the above-mentioned graph neural network, the recognition accuracy is improved while avoiding overfitting in the learning process.

[0036] In any of the above technical solutions, the neural network model includes an encoder-decoder, and the method also includes: determining feature points in key frame data; determining image blocks with the feature points as the center and different scales as the radius; and training the encoder-decoder based on the image blocks and the second pose data pair.

[0037] In this technical solution, feature points are extracted from key frame data so that image blocks can be extracted based on the feature points, so that the encoder-decoder can be trained using the extracted image blocks.

[0038] In the above process, the size and volume of the image blocks obtained after extraction are smaller than the key frame data before extraction. Therefore, when the image blocks are used for encoder-decoder training, the amount of training data required can be reduced.

[0039] In the above solution, since the amount of training data required during training is reduced, the training efficiency of the encoder-decoder can be improved.

[0040] In the above technical solution, different scales can be selected according to the needs of the encoder-decoder and are not limited here.

[0041] In any of the above technical solutions, the image block includes: an RGB image data block and a depth image data block; the encoder-decoder is trained based on the image block and the second pose data pair, including: superimposing the RGB image data block and the depth image data block in each key frame data to obtain a superimposed result; encoding the superimposed result to obtain an encoded result; processing the encoded result to obtain a query frame and a map frame; and the training is terminated when the similarity between the query frame and the map frame meets the preset requirements.

[0042] In this technical solution, a specific encoder-decoder training process is given, wherein the visual sensor is an RGBD visual sensor, and the data obtained using the RGBD visual sensor are RGB image data and depth image data, wherein R in the RGB image data is red, G is green, and B is blue.

[0043] In the above technical solution, the encoding result mentioned above is the position encoding of the data block.

[0044] In the above technical solution, the similarity is limited to meet the preset requirements to ensure that the trained encoder-decoder can find the attribute information in the target network node and match it with the visual data, so as to ensure the accuracy of the first pose data.

[0045] In the above technical solution, the similarity mentioned above can be represented by a calculation result based on the cosine similarity formula. When the calculation result is within a preset numerical range, it is considered that the similarity meets the preset requirements.

[0046] In the above technical solution, error optimization and parameter update can also be performed according to the true value of the similarity matrix to ensure the accuracy of the trained encoder-decoder.

[0047] In any of the above technical solutions, the encoder-decoder is trained using a sliding window method.

[0048] In this technical solution, a sliding window method is used for training, so as to reduce the amount of training of the encoder-decoder during the training process, and at the same time, improve the robustness of the training results.

[0049] In the above technical solution, considering that the pose query needs to take into account the correlation between the previous and next frames, we use a graph convolutional network to convolve the attributes of two adjacent nodes in the graph as the features of the edges connecting the nodes in the graph neural network, and connect the features of multiple nodes together to form a sliding window.

[0050] In the above technical solutions, in any of the above technical solutions, the encoding results are processed using a Transformer Encoder and a multi-layer perception network to obtain a query frame and a map frame.

[0051] In this technical solution, Transformer is a model structure that can be regarded as a black box, which includes an Encoder part and a Decoder part, where the Encoder is the encoding part and the Decoder is the decoding part.

[0052] In the above technical solution, the multilayer perceptron (MLP) is a feedforward artificial neural network model that maps multiple input data sets to a single output data set.

[0053] In any of the above technical solutions, the target network node is the network node with the highest similarity.

[0054] In this technical solution, the network node with the highest similarity is selected as the target network node so that the first pose data obtained can best match the acquired visual data and be the most accurate.

[0055] According to the second aspect of the present invention, the present invention provides a posture recovery device for a robot, the robot including a visual sensor, the posture recovery device including: an acquisition unit, used to obtain visual data collected by the visual sensor when the robot loses positioning information; a query unit, used to input the visual data into a preset neural network model, so that the preset neural network model encodes the visual data to obtain an encoding result, and compares the encoding result with the attribute information of the network node in the preset neural network model for similarity to obtain a comparison result, and determines the target network node based on the comparison result, wherein the attribute information of the target network node includes the first posture data; an output unit, used to output the first posture data.

[0056] The technical solution of the present application proposes a posture recovery device. A robot equipped with this device can quickly determine the robot's posture data based on the acquired visual data when the robot loses sensor data due to an emergency during mapping or when an initial position is required under known map navigation positioning conditions. This overcomes the problem of robot kidnapping and reduces the chance of the robot's navigation system crashing due to lack of posture data.

[0057] By analyzing a large amount of robot relocalization data and algorithms, it was found that sensor observation data in the environment (such as images, point clouds and other visual information) can form nonlinear constraints with the trajectory and pose graph generated when the map is established.

[0058] Based on this, the technical solution of the present application pre-trains the neural network model in order to obtain a trained preset neural network model, so that when positioning information is lost, the acquired visual data is sent to the preset neural network model, and the visual data is encoded and searched based on the trained preset neural network model, so as to efficiently regress the current posture of the robot.

[0059] Among them, since the encoding process of visual data occurs during the processing of the preset neural network model, there is no need to set up an additional encoder-decoder. The robot can directly send the visual data to the preset neural network model, which simplifies the data processing process.

[0060] In addition, by performing similarity comparison, the target network nodes obtained are matched with the visual data, ensuring the accuracy of the first pose data and the visual data.

[0061] It is worth noting that the posture data recovery process of the existing technical solution requires the robot to be controlled to move continuously in order to ensure that the motion equation and observation equation can be updated at all times. Since the robot is in a mobile state, it is necessary to execute the robot obstacle avoidance algorithm during the movement to prevent the robot from colliding with obstacles during the movement. The posture data recovery process of the present application can be implemented when the robot is stationary, thereby reducing the computational complexity of the posture data recovery process. At the same time, it also reduces the probability of losing positioning again during the posture data recovery process.

[0062] In addition, the posture recovery device proposed in this application also has the following additional technical features.

[0063] In the above technical solution, the query unit is also used to: construct a graph neural network; obtain historical visual data; determine the key frame data in the training data and the second pose data corresponding to the key frame based on the historical visual data; train the graph neural network based on the second pose data and the key frame data to obtain a preset neural network model.

[0064] In this technical solution, a training process for a preset neural network model is specifically provided. In this technical solution, since historical visual data all have corresponding pose data, the historical visual data is obtained so that training data can be determined based on the historical visual data. The training data includes corresponding second pose data and keyframe data. It can be understood that keyframe data, also known as a keyframe, can be understood as the skeleton of SLAM, which is a frame selected from a series of local ordinary frames as a representative local frame.

[0065] In one of the technical solutions, the second posture data can be understood as the posture data of the robot under the key frame.

[0066] In this technical solution, by selecting key frames and second pose data for model training, the time spent on training can be reduced and the training speed can be improved.

[0067] In this technical solution, graph neural network can be understood as a deep learning method based on graph structure, which mainly includes two parts, namely "graph" and "neural network", among which "graph" is the graph data structure in graph theory, and "neural network" is the deep learning NN structure.

[0068] In the above technical solution, historical visual data is processed based on the VINS-RGBD visual odometry calculation method to obtain key frame data and second pose data corresponding to each other.

[0069] In any of the above technical solutions, the key frame data includes target key frame data and remaining key frame data, and the query unit is specifically used to: project each pixel point in the target key frame data to the remaining key frame data according to the second posture data to obtain a similarity matrix; train the graph neural network according to the similarity matrix to obtain a preset neural network model.

[0070] In this technical solution, since the similarity matrix reflects the data relationship between different key frames, in the process of training the graph neural network using the similarity matrix, the preset neural network model obtained by training is no longer the training result between a single key frame, but can reflect the relationship between key frames, thereby realizing the fusion of attributes between network nodes and ensuring the accuracy of the first pose data output by the trained preset neural network model.

[0071] In the above technical solution, the target key frame data may be any key frame in the key frame data, and the remaining key frame data is the key frame data other than the target key frame data in the key frame data.

[0072] In any of the above technical solutions, the query unit is specifically used to: determine the coordinate value of each pixel point in the target key frame data projected on the remaining key frame data; determine the number of pixels of the target key frame data projected on the remaining key frame data based on the coordinate value; determine the intersection-and-union ratio of the target key frame data and the remaining key frame data based on the number of pixels; and determine the similarity matrix based on the intersection-and-union ratio.

[0073] In this technical solution, a specific determination scheme for the similarity matrix is ​​specifically given. In this scheme, the intersection-and-union ratio is calculated by counting the number of pixel points projected by the target key frame data on the remaining key frame data, and the similarity matrix is ​​determined using the intersection-and-union ratio.

[0074] In this process, in order to determine whether the pixel points in the target key frame data have projections on the remaining key frame data, the technical solution of the present application counts the coordinate values ​​of each pixel point in the target key frame data after projection. If the coordinate value is within the coordinate interval corresponding to the remaining key frame data, the pixel point is considered to be a valid pixel point, and if the coordinate value is outside the coordinate interval corresponding to the remaining key frame data, the pixel point is considered to be an invalid pixel point. By counting the number of valid pixel points as the number of pixels, in this solution, the number of pixels can be understood as the number of pixels overlapping between the target key frame data and the remaining key frame data.

[0075] In this technical solution, the sum of the number of pixels of the target key frame data and the number of pixels of the remaining key frame data is also determined, and further, the difference between the sum and the number of pixels is determined, and the ratio of the number of pixels to the difference is used as the intersection-union ratio.

[0076] The similarity matrix can be obtained by selecting all target key frames in the key frame data.

[0077] In one possible design, each frame of the image is projected by the pose to obtain a similarity matrix. The specific projection method is as follows:

[0078] p k+1 '=Tp k

[0079] Among them, T is the posture matrix, P k is the target key frame data, p k+1 ' is the remaining key frame data.

[0080] In any of the above technical solutions, the query unit is specifically used to: use the pose node corresponding to the key frame data as the network node; use the second pose data and the encoded visual data as the attribute information of the network node; connect all the nodes to each other to obtain a graph neural network.

[0081] In this technical solution, a specific method for constructing a graph neural network is given. In this method, the network node selects the pose node corresponding to the key frame data, and uses the encoded visual data and the second pose data as attribute information, and connects all nodes to achieve full connection.

[0082] In the above process, due to the introduction of the above-mentioned graph neural network, the recognition accuracy is improved while avoiding overfitting in the learning process.

[0083] In any of the above technical solutions, the neural network model includes an encoder-decoder and a query unit, and is specifically used to: determine the feature points in the key frame data; determine the image blocks with the feature points as the center and the radius of different scales; and train the encoder-decoder based on the image blocks and the second pose data.

[0084] In this technical solution, feature points are extracted from key frame data so that image blocks can be extracted based on the feature points, so that the encoder-decoder can be trained using the extracted image blocks.

[0085] In the above process, the size and volume of the image blocks obtained after extraction are smaller than the key frame data before extraction. Therefore, when the image blocks are used for encoder-decoder training, the amount of training data required can be reduced.

[0086] In the above solution, since the amount of training data required during training is reduced, the training efficiency of the encoder-decoder can be improved.

[0087] In the above technical solution, different scales can be selected according to the needs of the encoder-decoder and are not limited here.

[0088] In any of the above technical solutions, the image block includes: an RGB image data block and a depth image data block; the query unit is specifically used to: superimpose the RGB image data block and the depth image data block in each key frame data to obtain a superposition result; encode the superposition result to obtain an encoding result; process the encoding result to obtain a query frame and a map frame; and end the training when the similarity between the query frame and the map frame meets the preset requirements.

[0089] In this technical solution, a specific encoder-decoder training process is given, wherein the visual sensor is an RGBD visual sensor, and the data obtained using the RGBD visual sensor are RGB image data and depth image data, wherein R in the RGB image data is red, G is green, and B is blue.

[0090] In the above technical solution, the encoding result mentioned above is the position encoding of the data block.

[0091] In the above technical solution, the similarity is limited to meet the preset requirements to ensure that the trained encoder-decoder can find the attribute information in the target network node and match it with the visual data, so as to ensure the accuracy of the first pose data.

[0092] In the above technical solution, the similarity mentioned above can be represented by a calculation result based on the cosine similarity formula. When the calculation result is within a preset numerical range, it is considered that the similarity meets the preset requirements.

[0093] In the above technical solution, error optimization and parameter update can also be performed according to the true value of the similarity matrix to ensure the accuracy of the trained encoder-decoder.

[0094] In any of the above technical solutions, the query unit is further configured to train the encoder-decoder using a sliding window method.

[0095] In this technical solution, a sliding window method is used for training, so as to reduce the amount of training of the encoder-decoder during the training process, and at the same time, improve the robustness of the training results.

[0096] In the above technical solution, considering that the pose query needs to take into account the correlation between the previous and next frames, we use a graph convolutional network to convolve the attributes of two adjacent nodes in the graph as the features of the edges connecting the nodes in the graph neural network, and connect the features of multiple nodes together to form a sliding window.

[0097] In any of the above technical solutions, the query unit is further configured to: process the encoding result using a Transformer Encoder and a multi-layer perception network to obtain a query frame and a map frame.

[0098] In this technical solution, Transformer is a model structure that can be regarded as a black box, which includes an Encoder part and a Decoder part, where the Encoder is the encoding part and the Decoder is the decoding part.

[0099] In the above technical solution, the multilayer perceptron (MLP) is a feedforward artificial neural network model that maps multiple input data sets to a single output data set.

[0100] In any of the above technical solutions, the comparison result is the attribute information of the network node with the highest similarity.

[0101] In this technical solution, the network node with the highest similarity is selected as the target network node so that the first pose data obtained can best match the acquired visual data and be the most accurate.

[0102] According to the third aspect of the present invention, the present invention provides a posture recovery device, comprising: a controller and a memory, wherein the memory stores a program or instruction, and the controller implements the steps of any of the above methods when executing the program or instruction in the memory.

[0103] According to a fourth aspect of the present invention, the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of any of the above methods are implemented.

[0104] According to a fifth aspect of the present invention, the present invention provides a robot comprising: any one of the posture recovery devices described above; or the readable storage medium described above.

[0105] In the above technical solution, the robot includes a sweeping robot.

[0106] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments with reference to the following drawings, in which:

[0108] Figure 1 1 shows a flow chart of a posture recovery method according to an embodiment of the present invention;

[0109] Figure 2 A schematic diagram showing the construction of a graph neural network in an embodiment of the present invention is shown;

[0110] Figure 3 A schematic diagram showing feature points and image blocks in an embodiment of the present invention is shown;

[0111] Figure 4 A schematic diagram showing how the encoding result is processed to obtain a query frame and a map frame in an embodiment of the present invention;

[0112] Figure 5 A schematic diagram of determining a sliding window in an embodiment of the present invention is shown;

[0113] Figure 6 The figure shows the overall process of posture recovery in an embodiment of the present invention;

[0114] Figure 7 A schematic block diagram of a posture recovery device according to an embodiment of the present invention is shown;

[0115] Figure 8 A schematic block diagram of a posture recovery device in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0116] In order to more clearly understand the above aspects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features therein can be combined with each other in the absence of conflict.

[0117] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0118] Refer to the following Figures 1 to 8 The present invention describes a posture recovery method, an apparatus, a readable storage medium, and a robot according to some embodiments of the present invention.

[0119] like Figure 1 As shown, according to a first aspect of the present invention, the present invention provides a posture recovery method for a robot, the robot including a visual sensor, and the posture recovery method including:

[0120] Step 102: When the robot loses positioning information, obtain visual data collected by the visual sensor;

[0121] Step 104: Input the visual data into a preset neural network model so that the preset neural network model encodes the visual data to obtain an encoding result, and compare the encoding result with the attribute information of the network node in the preset neural network model for similarity to obtain a comparison result. Based on the comparison result, determine the target network node, wherein the attribute information of the target network node includes the first pose data;

[0122] Step 106: output the first pose data.

[0123] The design of this application proposes a posture recovery method. By running this recovery method, when the sensor data is lost due to an emergency during the robot mapping process or when the initial position is required under the condition of known map navigation positioning, the robot's posture data can be quickly determined based on the acquired visual data. This overcomes the existence of the robot kidnapping problem and reduces the chance of the robot's navigation system crashing due to the lack of posture data.

[0124] SLAM (simultaneous localization and mapping), also known as CML (Concurrent Mapping and Localization), is a technology that allows for simultaneous positioning and mapping, or concurrent mapping and localization. The problem can be described as follows: Given a robot placed in an unknown environment and at an unknown location, is there a way to enable the robot to gradually create a complete map of the environment as it moves? A consistent map means being able to reach every accessible corner of the room without obstacles.

[0125] By analyzing a large amount of robot relocalization data and algorithms, it was found that sensor observation data in the environment (such as images, point clouds and other visual information) can form nonlinear constraints with the trajectory and pose graph generated when the map is established.

[0126] Based on this, the design of this application pre-trains the neural network model in order to obtain a trained preset neural network model, so that when positioning information is lost, the acquired visual data is sent to the preset neural network model, and the visual data is encoded and searched based on the trained preset neural network model, so as to efficiently regress the current posture of the robot.

[0127] Among them, since the encoding process of visual data occurs during the processing of the preset neural network model, there is no need to set up an additional encoder-decoder. The robot can directly send the visual data to the preset neural network model, which simplifies the data processing process.

[0128] In addition, by performing similarity comparison, the target network nodes obtained are matched with the visual data, ensuring the accuracy of the first pose data and the visual data.

[0129] It is worth noting that the posture data recovery process of the existing design requires the robot to be controlled to move continuously so as to ensure that the motion equation and observation equation can be updated at all times. Since the robot is in a mobile state, the robot needs to execute the robot obstacle avoidance algorithm during the movement to prevent the robot from colliding with obstacles during the movement. The posture data recovery process of the present application can be implemented when the robot is stationary, thereby reducing the computational complexity of the posture data recovery process. At the same time, it also reduces the probability of losing positioning again during the posture data recovery process.

[0130] In the above design, the solution can be applied to robot topology navigation and planning, as well as blueprint design and verification in the architectural field, and can be helpful for 3D map display. Furthermore, in terms of sensor requirements, this solution only requires visual sensors, such as RGBD vision sensors, which has low sensor requirements and a wide range of applications.

[0131] In one possible design, the usage scenarios of the above solution are limited to regular indoor scenes and outdoor scenes in urban environments, so as to improve the accuracy of the first pose data.

[0132] In the above design, the posture recovery method also includes: constructing a graph neural network; obtaining historical visual data; determining the key frame data in the training data and the second posture data corresponding to the key frame based on the historical visual data; training the graph neural network based on the second posture data and the key frame data to obtain a preset neural network model.

[0133] In this design, a training process of a preset neural network model is specifically given. In this design, since historical visual data all have corresponding pose data, the historical visual data is obtained so that the training data can be determined based on the historical visual data. The training data includes corresponding second pose data and key frame data. It can be understood that the key frame data is also a key frame, which can be understood as the skeleton of SLAM. It is a frame selected from a series of local ordinary frames as a representative of the local frame.

[0134] In one of the designs, the second pose data can be understood as the pose data of the robot under the key frame.

[0135] In this design, by selecting key frames and second pose data for model training, the time spent on training can be reduced and the training speed can be improved.

[0136] In this design, graph neural network can be understood as a deep learning method based on graph structure, which mainly consists of two parts, namely "graph" and "neural network", among which "graph" is the graph data structure in graph theory, and "neural network" is the deep learning NN structure.

[0137] In the above design, the historical visual data is processed based on the VINS-RGBD visual odometry calculation method to obtain the corresponding key frame data and second pose data.

[0138] In any of the above designs, the key frame data includes target key frame data and remaining key frame data. The graph neural network is trained based on the second pose data and the key frame data to obtain a preset neural network model, including: projecting each pixel point in the target key frame data to the remaining key frame data based on the second pose data to obtain a similarity matrix; and training the graph neural network based on the similarity matrix to obtain a preset neural network model.

[0139] In this design, since the similarity matrix reflects the data relationship between different keyframes, in the process of training the graph neural network using the similarity matrix, the preset neural network model obtained by training is no longer the training result between a single keyframe, but can reflect the relationship between keyframes, thereby realizing the fusion of attributes between network nodes and ensuring the accuracy of the first pose data output by the trained preset neural network model.

[0140] In the above design, the target key frame data may be any key frame in the key frame data, and the remaining key frame data is the key frame data other than the target key frame data in the key frame data.

[0141] For example, the key frame data contains 4 key frames, namely, key frame 1, key frame 2, key frame 3 and key frame 4. When key frame 1 is the target key frame data, the remaining key frame data are key frame 2, key frame 3 and key frame 4. The posture data of key frame 1 is projected onto key frame 2, key frame 3 and key frame 4 to obtain three projection results. Similarly, when the target key frame data is selected as other key frames, other projection results can be obtained. According to the projection results, a similarity matrix can be obtained so that the model can be trained using the similarity matrix.

[0142] In any of the above designs, each pixel point in the target key frame data is projected onto the remaining key frame data according to the second posture data to obtain a similarity matrix, which specifically includes: determining the coordinate value of each pixel point in the target key frame data projected on the remaining key frame data; determining the number of pixels of the target key frame data projected on the remaining key frame data according to the coordinate value; determining the intersection-union ratio of the target key frame data and the remaining key frame data according to the number of pixels; and determining the similarity matrix according to the intersection-union ratio.

[0143] In this design, a specific scheme for determining the similarity matrix is ​​given. In this scheme, the intersection-and-union ratio is calculated by counting the number of pixel points projected by the target key frame data on the remaining key frame data, and the similarity matrix is ​​determined using the intersection-and-union ratio.

[0144] In this process, in order to determine whether the pixel points in the target key frame data have projections on the remaining key frame data, the design of this application counts the coordinate values ​​of each pixel point in the target key frame data after projection. If the coordinate value is within the coordinate interval corresponding to the remaining key frame data, the pixel point is considered to be a valid pixel point, and if the coordinate value is outside the coordinate interval corresponding to the remaining key frame data, the pixel point is considered to be an invalid pixel point. By counting the number of valid pixel points as the number of pixels, in this scheme, the number of pixels can be understood as the number of pixels overlapping between the target key frame data and the remaining key frame data.

[0145] In this design, the sum of the number of pixels of the target key frame data and the number of pixels of the remaining key frame data is also determined. Further, the difference between the sum and the number of pixels is determined, and the ratio of the number of pixels to the difference is used as the intersection-union ratio.

[0146] For example, the key frame data includes 4 key frames, namely, key frame 1, key frame 2, key frame 3 and key frame 4. When key frame 1 is the target key frame data, the remaining key frame data are key frame 2, key frame 3 and key frame 4. When key frame 1 is projected onto key frame 2, the number of pixels in key frame 2 is 10, the total number of pixels in key frame 1 is 30, and the total number of pixels in key frame 2 is 20. The intersection-over-union ratio is 10 / (30+20)=0.2.

[0147] The similarity matrix can be obtained by selecting all target key frames in the key frame data.

[0148] In one possible design, each frame of the image is projected by the pose to obtain a similarity matrix. The specific projection method is as follows:

[0149] p k+1 '=Tp k

[0150] Among them, T is the posture matrix, P k is the target key frame data, p k+1 ' is the remaining key frame data.

[0151] In any of the above designs, a graph neural network is constructed, including: taking the pose nodes corresponding to the key frame data as network nodes; taking the second pose data and the encoded visual data as attribute information of the network nodes; and connecting all the nodes to each other to obtain a graph neural network.

[0152] In this design, a specific method for constructing a graph neural network is given, such as Figure 2 As shown, in this method, the network node selects the pose node corresponding to the key frame data, and uses the encoded visual data and the second pose data as attribute information, and connects all nodes to achieve full connection.

[0153] in, Figure 2 The left side of the middle image is the second pose data, which can be understood as a pose graph, and the right side is the graph neural network, which is also a fully connected graph.

[0154] In the above process, due to the introduction of the above-mentioned graph neural network, the recognition accuracy is improved while avoiding overfitting in the learning process.

[0155] In any of the above designs, the neural network model includes an encoder-decoder, and the method further includes: determining feature points in key frame data; determining image blocks with different scales as radii centered on the feature points; and training the encoder-decoder based on the image blocks and the second pose data pair.

[0156] In this design, if Figure 3 As shown, by extracting feature points from key frame data, image blocks can be extracted based on the feature points, so that the encoder-decoder can be trained using the extracted image blocks.

[0157] In the above process, the size and volume of the image blocks obtained after extraction are smaller than the key frame data before extraction. Therefore, when the image blocks are used for encoder-decoder training, the amount of training data required can be reduced.

[0158] In the above solution, since the amount of training data required during training is reduced, the training efficiency of the encoder-decoder can be improved.

[0159] In the above design, different scales can be selected according to the needs of the encoder-decoder and are not limited here.

[0160] In the above technical solution, since the sampling time of different key frame data is different, the corresponding posture data is also at different times. Based on this, the posture data at different times can be used to construct an error function, and finally the optimal parameters of the encoder-decoder can be regressed.

[0161] In this technical solution, the relative distance between the posture data corresponding to different key frame data in the training set is calculated, and whether they have arrived at the same scene is determined based on the calculation results. When the distance between the posture data corresponding to different key frame data is less than the distance threshold, it is considered that the current two key frame data have arrived at the same scene, and the encoder-decoder is trained based on this.

[0162] In any of the above designs, if Figure 4 As shown, the image block includes: an RGB image data block and a depth image data block; the encoder-decoder is trained according to the image block and the second pose data pair, including: superimposing the RGB image data block and the depth image data block in each key frame data to obtain a superimposed result; encoding the superimposed result to obtain an encoded result; processing the encoded result to obtain a query frame and a map frame; and the training ends when the similarity between the query frame and the map frame meets the preset requirements.

[0163] In this design, the encoder-decoder training process is specifically given, where the visual sensor is an RGBD visual sensor, and the data obtained using the RGBD visual sensor are RGB image data and depth image data, where R in the RGB image data is red, G is green, and B is blue.

[0164] In the above design, the encoding result above is the position encoding of the data block.

[0165] In the above design, the similarity is limited to meet the preset requirements to ensure that the trained encoder-decoder can find the attribute information in the target network node and match it with the visual data, so as to ensure the accuracy of the first pose data.

[0166] In the above design, the similarity mentioned above can be represented by the calculation result based on the cosine similarity formula. When the calculation result is within a preset numerical range, it is considered that the similarity meets the preset requirements.

[0167] In the above design, error optimization and parameter update can also be performed according to the true value of the similarity matrix to ensure the accuracy of the trained encoder-decoder.

[0168] In any of the above designs, the encoder-decoder is trained using a sliding window approach.

[0169] In this design, a sliding window approach is adopted for training, so as to reduce the amount of training of the encoder-decoder during the training process and improve the robustness of the training results.

[0170] In the above design, if Figure 5 As shown in the figure, considering that the pose query needs to take into account the correlation between the previous and next frames, we use the graph convolutional network to convolve the attributes of two adjacent nodes in the graph as the features of the edges connecting the nodes in the graph neural network, and connect the features of multiple nodes together to form a sliding window.

[0171] In the above designs, in any of the above designs, the encoding results are processed using a Transformer Encoder and a multi-layer perception network to obtain a query frame and a map frame.

[0172] In this design, Transformer is a model structure that can be regarded as a black box, which includes an Encoder part and a Decoder part, where the Encoder is the encoding part and the Decoder is the decoding part.

[0173] In the above design, the multilayer perceptron (MLP) is a feedforward artificial neural network model that maps multiple input data sets to a single output data set.

[0174] In any of the above designs, the target network node is the network node with the highest similarity.

[0175] In this design, the network node with the highest similarity is selected as the target network node so that the first pose data obtained can best match the acquired visual data and be the most accurate.

[0176] In one embodiment, Figure 6 As shown in Figure 2, the overall process of posture recovery is as follows:

[0177] Step 602, sensor image input;

[0178] Step 604: image preprocessing;

[0179] Step 606, key frame;

[0180] Step 608, pose graph;

[0181] Step 610, calculating a similarity matrix;

[0182] Step 612: extracting image blocks;

[0183] Step 614: Convolutional neural network encodes the image.

[0184] Step 616: The encoder-decoder performs encoding and decoding;

[0185] Step 618, similarity prediction and back propagation;

[0186] Step 620, graph neural network adjacent node graph convolution;

[0187] Step 622, the convolved attributes are aggregated with the node attributes;

[0188] Step 624: Graph neural network pose regression and training;

[0189] Step 626: pose regression and network training.

[0190] Among them, the sensor image input is also the historical visual data in this application, wherein the key frame is the key frame data in the training set, and the pose graph is the second pose data.

[0191] In one embodiment, the present invention provides a posture recovery device for a robot, wherein the robot includes a visual sensor, such as Figure 7 As shown, the posture recovery device 700 includes: an acquisition unit 702, which is used to obtain visual data collected by the visual sensor when the robot loses positioning information; a query unit 704, which is used to input the visual data into a preset neural network model so that the preset neural network model encodes the visual data to obtain an encoding result, and compares the encoding result with the attribute information of the network node in the preset neural network model to obtain a comparison result, and determines the target network node based on the comparison result, wherein the attribute information of the target network node includes the first posture data; an output unit 706, which is used to output the first posture data.

[0192] The design of this application proposes a posture recovery device 700. A robot equipped with this device can quickly determine the robot's posture data based on the acquired visual data when the robot loses sensor data due to an emergency during mapping or requires an initial position under known map navigation positioning conditions. This overcomes the problem of robot kidnapping and reduces the chance of the robot's navigation system crashing due to lack of posture data.

[0193] SLAM (simultaneous localization and mapping), also known as CML (Concurrent Mapping and Localization), is a technology that allows for simultaneous positioning and mapping, or concurrent mapping and localization. The problem can be described as follows: Given a robot placed in an unknown environment and at an unknown location, is there a way to enable the robot to gradually create a complete map of the environment as it moves? A consistent map means being able to reach every accessible corner of the room without obstacles.

[0194] By analyzing a large amount of robot relocalization data and algorithms, it was found that sensor observation data in the environment (such as images, point clouds and other visual information) can form nonlinear constraints with the trajectory and pose graph generated when the map is established.

[0195] Based on this, the design of this application pre-trains the neural network model in order to obtain a trained preset neural network model, so that when positioning information is lost, the acquired visual data is sent to the preset neural network model, and the visual data is encoded and searched based on the trained preset neural network model, so as to efficiently regress the current posture of the robot.

[0196] Among them, since the encoding process of visual data occurs during the processing of the preset neural network model, there is no need to set up an additional encoder-decoder. The robot can directly send the visual data to the preset neural network model, which simplifies the data processing process.

[0197] In addition, by performing similarity comparison, the target network nodes obtained are matched with the visual data, ensuring the accuracy of the first pose data and the visual data.

[0198] It is worth noting that the posture data recovery process of the existing design requires the robot to be controlled to move continuously so as to ensure that the motion equation and observation equation can be updated at all times. Since the robot is in a mobile state, the robot needs to execute the robot obstacle avoidance algorithm during the movement to prevent the robot from colliding with obstacles during the movement. The posture data recovery process of the present application can be implemented when the robot is stationary, thereby reducing the computational complexity of the posture data recovery process. At the same time, it also reduces the probability of losing positioning again during the posture data recovery process.

[0199] In the above design, the query unit 704 is also used to: construct a graph neural network; obtain historical visual data; determine the key frame data in the training data and the second pose data corresponding to the key frame based on the historical visual data; train the graph neural network based on the second pose data and the key frame data to obtain a preset neural network model.

[0200] In this design, a training process of a preset neural network model is specifically given. In this design, since historical visual data all have corresponding pose data, the historical visual data is obtained so that the training data can be determined based on the historical visual data. The training data includes corresponding second pose data and key frame data. It can be understood that the key frame data is also a key frame, which can be understood as the skeleton of SLAM. It is a frame selected from a series of local ordinary frames as a representative of the local frame.

[0201] In one of the designs, the second pose data can be understood as the pose data of the robot under the key frame.

[0202] In this design, by selecting key frames and second pose data for model training, the time spent on training can be reduced and the training speed can be improved.

[0203] In this design, graph neural network can be understood as a deep learning method based on graph structure, which mainly consists of two parts, namely "graph" and "neural network", among which "graph" is the graph data structure in graph theory, and "neural network" is the deep learning NN structure.

[0204] In the above design, the historical visual data is processed based on the VINS-RGBD visual odometry calculation method to obtain the corresponding key frame data and second pose data.

[0205] In any of the above designs, the key frame data includes target key frame data and remaining key frame data, and the query unit 704 is specifically used to: project each pixel point in the target key frame data to the remaining key frame data according to the second posture data to obtain a similarity matrix; train the graph neural network according to the similarity matrix to obtain a preset neural network model.

[0206] In this design, since the similarity matrix reflects the data relationship between different keyframes, in the process of training the graph neural network using the similarity matrix, the preset neural network model obtained by training is no longer the training result between a single keyframe, but can reflect the relationship between keyframes, thereby realizing the fusion of attributes between network nodes and ensuring the accuracy of the first pose data output by the trained preset neural network model.

[0207] In the above design, the target key frame data may be any key frame in the key frame data, and the remaining key frame data is the key frame data other than the target key frame data in the key frame data.

[0208] For example, the key frame data contains 4 key frames, namely, key frame 1, key frame 2, key frame 3 and key frame 4. When key frame 1 is the target key frame data, the remaining key frame data are key frame 2, key frame 3 and key frame 4. The posture data of key frame 1 is projected onto key frame 2, key frame 3 and key frame 4 to obtain three projection results. Similarly, when the target key frame data is selected as other key frames, other projection results can be obtained. According to the projection results, a similarity matrix can be obtained so that the model can be trained using the similarity matrix.

[0209] In any of the above designs, the query unit 704 is further specifically used to: determine the coordinate value of each pixel point in the target key frame data projected on the remaining key frame data; determine the number of pixels of the target key frame data projected on the remaining key frame data based on the coordinate value; determine the intersection-and-union ratio of the target key frame data and the remaining key frame data based on the number of pixels; and determine the similarity matrix based on the intersection-and-union ratio.

[0210] In this design, a specific scheme for determining the similarity matrix is ​​given. In this scheme, the intersection-and-union ratio is calculated by counting the number of pixel points projected by the target key frame data on the remaining key frame data, and the similarity matrix is ​​determined using the intersection-and-union ratio.

[0211] In this process, in order to determine whether the pixel points in the target key frame data have projections on the remaining key frame data, the design of this application counts the coordinate values ​​of each pixel point in the target key frame data after projection. If the coordinate value is within the coordinate interval corresponding to the remaining key frame data, the pixel point is considered to be a valid pixel point, and if the coordinate value is outside the coordinate interval corresponding to the remaining key frame data, the pixel point is considered to be an invalid pixel point. By counting the number of valid pixel points as the number of pixels, in this scheme, the number of pixels can be understood as the number of pixels overlapping between the target key frame data and the remaining key frame data.

[0212] In this design, the sum of the number of pixels of the target key frame data and the number of pixels of the remaining key frame data is also determined. Further, the difference between the sum and the number of pixels is determined, and the ratio of the number of pixels to the difference is used as the intersection-union ratio.

[0213] For example, the key frame data includes 4 key frames, namely, key frame 1, key frame 2, key frame 3 and key frame 4. When key frame 1 is the target key frame data, the remaining key frame data are key frame 2, key frame 3 and key frame 4. When key frame 1 is projected onto key frame 2, the number of pixels in key frame 2 is 10, the total number of pixels in key frame 1 is 30, and the total number of pixels in key frame 2 is 20. The intersection-over-union ratio is 10 / (30+20)=0.2.

[0214] The similarity matrix can be obtained by selecting all target key frames in the key frame data.

[0215] In one possible design, each frame of the image is projected by the pose to obtain a similarity matrix. The specific projection method is as follows:

[0216] p k+1 '=Tp k

[0217] Among them, T is the posture matrix, P kis the target key frame data, p k+1 ' is the remaining key frame data.

[0218] In any of the above designs, the query unit 704 is specifically used to: use the pose node corresponding to the key frame data as the network node; use the second pose data and the encoded visual data as the attribute information of the network node; and connect all the nodes to each other to obtain a graph neural network.

[0219] In this design, a specific method for constructing a graph neural network is given. In this method, the network node selects the pose node corresponding to the key frame data, and uses the encoded visual data and the second pose data as attribute information, and connects all nodes to achieve full connection.

[0220] In the above process, due to the introduction of the above-mentioned graph neural network, the recognition accuracy is improved while avoiding overfitting in the learning process.

[0221] In any of the above designs, the neural network model includes an encoder-decoder and a query unit 704, which is specifically used to: determine feature points in key frame data; determine image blocks with feature points as the center and different scales as the radius; and train the encoder-decoder based on the image blocks and the second pose data.

[0222] In this design, feature points are extracted from key frame data so that image blocks can be extracted based on the feature points, so that the encoder-decoder can be trained using the extracted image blocks.

[0223] In the above process, the size and volume of the image blocks obtained after extraction are smaller than the key frame data before extraction. Therefore, when the image blocks are used for encoder-decoder training, the amount of training data required can be reduced.

[0224] In the above solution, since the amount of training data required during training is reduced, the training efficiency of the encoder-decoder can be improved.

[0225] In the above design, different scales can be selected according to the needs of the encoder-decoder and are not limited here.

[0226] In any of the above designs, the image block includes: an RGB image data block and a depth image data block; the query unit 704 is specifically used to: superimpose the RGB image data block and the depth image data block in each key frame data to obtain a superposition result; encode the superposition result to obtain an encoding result; process the encoding result to obtain a query frame and a map frame; when the similarity between the query frame and the map frame meets the preset requirements, the training is terminated.

[0227] In this design, the encoder-decoder training process is specifically given, where the visual sensor is an RGBD visual sensor, and the data obtained using the RGBD visual sensor are RGB image data and depth image data, where R in the RGB image data is red, G is green, and B is blue.

[0228] In the above design, the encoding result above is the position encoding of the data block.

[0229] In the above design, the similarity is limited to meet the preset requirements to ensure that the trained encoder-decoder can find the attribute information in the target network node and match it with the visual data, so as to ensure the accuracy of the first pose data.

[0230] In the above design, the similarity mentioned above can be represented by the calculation result based on the cosine similarity formula. When the calculation result is within a preset numerical range, it is considered that the similarity meets the preset requirements.

[0231] In the above design, error optimization and parameter update can also be performed according to the true value of the similarity matrix to ensure the accuracy of the trained encoder-decoder.

[0232] In any of the above designs, the query unit 704 is further configured to train the encoder-decoder using a sliding window approach.

[0233] In this design, a sliding window approach is adopted for training, so as to reduce the amount of training of the encoder-decoder during the training process and improve the robustness of the training results.

[0234] In the above design, considering that the pose query needs to take into account the correlation between the previous and next frames, we use the graph convolutional network to convolve the attributes of two adjacent nodes in the graph as the features of the edges connecting the nodes in the graph neural network, and connect the features of multiple nodes together to form a sliding window.

[0235] In any of the above designs, the query unit 704 is further configured to: process the encoding result using a Transformer Encoder and a multi-layer perception network to obtain a query frame and a map frame.

[0236] In this design, Transformer is a model structure that can be regarded as a black box, which includes an Encoder part and a Decoder part, where the Encoder is the encoding part and the Decoder is the decoding part.

[0237] In the above design, the multilayer perceptron (MLP) is a feedforward artificial neural network model that maps multiple input data sets to a single output data set.

[0238] In any of the above designs, the comparison result is the attribute information of the network node with the highest similarity.

[0239] In this design, the network node with the highest similarity is selected as the target network node so that the first pose data obtained can best match the acquired visual data and be the most accurate.

[0240] In one embodiment, Figure 8 As shown, the present invention provides a posture recovery device 800, including: a controller 802 and a memory 804, wherein the memory 804 stores a program or instruction, and the controller 802 implements the steps of any of the above methods when executing the program or instruction in the memory 804.

[0241] In one embodiment, the present invention provides a readable storage medium storing a program or instruction. When the program or instruction is executed by a processor, the steps of any of the above methods are implemented.

[0242] In one embodiment, the present invention provides a robot comprising: any one of the posture recovery devices described above; or the readable storage medium described above.

[0243] The design of this application proposes a robot that can quickly determine the robot's posture data based on the acquired visual data when the robot loses sensor data due to an emergency during mapping or when an initial position is required under known map navigation positioning conditions. This overcomes the problem of robot kidnapping and reduces the chance of the robot's navigation system crashing due to lack of posture data.

[0244] SLAM (simultaneous localization and mapping), also known as CML (Concurrent Mapping and Localization), is a technology that allows for simultaneous positioning and mapping, or concurrent mapping and localization. The problem can be described as follows: Given a robot placed in an unknown environment and at an unknown location, is there a way to enable the robot to gradually create a complete map of the environment as it moves? A consistent map means being able to reach every accessible corner of the room without obstacles.

[0245] By analyzing a large amount of robot relocalization data and algorithms, it was found that sensor observation data in the environment (such as images, point clouds and other visual information) can form nonlinear constraints with the trajectory and pose graph generated when the map is established.

[0246] Based on this, the design of this application pre-trains the neural network model in order to obtain a trained preset neural network model, so that when positioning information is lost, the acquired visual data is sent to the preset neural network model, and the visual data is encoded and searched based on the trained preset neural network model, so as to efficiently regress the current posture of the robot.

[0247] Among them, since the encoding process of visual data occurs during the processing of the preset neural network model, there is no need to set up an additional encoder-decoder. The robot can directly send the visual data to the preset neural network model, which simplifies the data processing process.

[0248] In addition, by performing similarity comparison, the target network nodes obtained are matched with the visual data, ensuring the accuracy of the first pose data and the visual data.

[0249] It is worth noting that the posture data recovery process of the existing design requires the robot to be controlled to move continuously so as to ensure that the motion equation and observation equation can be updated at all times. Since the robot is in a mobile state, the robot needs to execute the robot obstacle avoidance algorithm during the movement to prevent the robot from colliding with obstacles during the movement. The posture data recovery process of the present application can be implemented when the robot is stationary, thereby reducing the computational complexity of the posture data recovery process. At the same time, it also reduces the probability of losing positioning again during the posture data recovery process.

[0250] In the above design, the solution can be applied to robot topology navigation and planning, as well as blueprint design and verification in the architectural field, and can be helpful for 3D map display. Furthermore, in terms of sensor requirements, this solution only requires visual sensors, such as RGBD vision sensors, which has low sensor requirements and a wide range of applications.

[0251] In one possible design, the usage scenarios of the above solution are limited to regular indoor scenes and outdoor scenes in urban environments, so as to improve the accuracy of the first pose data.

[0252] In one possible design, the memory may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0253] In the above design, the robot includes a sweeping robot.

[0254] In the description of the present invention, the term "plurality" refers to two or more than two. Unless otherwise expressly defined, the orientations or positional relationships indicated by the terms "upper" and "lower" are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention. The terms "connect," "install," and "fix" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0255] In the description of the present invention, the terms "one embodiment," "some embodiments," "specific embodiments," etc., mean that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In the present invention, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0256] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A posture recovery method for a robot, characterized in that: The robot includes a visual sensor, and the posture recovery method includes: When the robot loses positioning information, obtaining visual data collected by the visual sensor; Inputting the visual data into a preset neural network model so that the preset neural network model encodes the visual data to obtain an encoding result, and performing a similarity comparison between the encoding result and attribute information of a network node in the preset neural network model to obtain a comparison result, and determining a target network node based on the comparison result, wherein the attribute information of the target network node includes first pose data; outputting the first posture data; The posture recovery method further includes: Build graph neural networks; Acquire historical visual data; Determine key frame data in the training data and second pose data corresponding to the key frame according to the historical visual data; Training the graph neural network according to the second pose data and key frame data to obtain the preset neural network model; The key frame data includes target key frame data and remaining key frame data, the target key frame data is any key frame in the key frame data, and the remaining key frame data is the key frame data in the key frame data except the target key frame data.

2. The posture recovery method according to claim 1, characterized in that: The key frame data includes target key frame data and remaining key frame data. The step of training the graph neural network according to the second pose data and the key frame data to obtain the preset neural network model includes: Projecting each pixel point in the target key frame data onto the remaining key frame data according to the second posture data to obtain a similarity matrix; The graph neural network is trained according to the similarity matrix to obtain the preset neural network model.

3. The posture recovery method according to claim 2, characterized in that: Based on the second pose data, each pixel in the target key frame data is projected onto the remaining key frame data to obtain a similarity matrix, which specifically includes: Determine the coordinate value of each pixel point in the target key frame data projected on the remaining key frame data; Determining the number of pixels of the target key frame data projected in the remaining key frame data according to the coordinate values; Determine an intersection-over-union ratio between target key frame data and remaining key frame data according to the number of pixels; The similarity matrix is ​​determined according to the intersection-over-union ratio.

4. The posture recovery method according to claim 1, characterized in that: Build a graph neural network, including: Using the pose node corresponding to the key frame data as the network node; Using the second pose data and the encoded visual data as attribute information of the network node; All nodes are connected to each other to obtain the graph neural network.

5. The posture recovery method according to any one of claims 1 to 4, characterized in that: The neural network model includes an encoder-decoder, and the posture recovery method further includes: Determining feature points in the key frame data; Determine an image block with the feature point as the center and different scales as the radius; An encoder-decoder is trained based on the image patch and the second pose data pair.

6. The posture recovery method according to claim 5, characterized in that: The image block includes: an RGB image data block and a depth image data block; Training an encoder-decoder based on the image block and the second pose data pair, comprising: Superimposing the RGB image data block and the depth image data block in each of the key frame data to obtain a superposition result; Encoding the superposition result to obtain an encoding result; Processing the encoding result to obtain a query frame and a map frame; When the similarity between the query frame and the map frame meets a preset requirement, the training is terminated.

7. The posture recovery method according to claim 6, characterized in that: The encoder-decoder is trained using a sliding window approach.

8. The posture recovery method according to claim 6, characterized in that: The encoding result is processed using a Transformer Encoder and a multi-layer perception network to obtain the query frame and the map frame.

9. The posture recovery method according to any one of claims 1 to 4, characterized in that: The target network node is the network node with the highest similarity.

10. A posture recovery device for a robot, characterized in that: The robot includes a visual sensor, and the posture recovery device includes: an acquisition unit, configured to acquire visual data collected by the visual sensor when the robot loses positioning information; a query unit, configured to input the visual data into a preset neural network model so that the preset neural network model encodes the visual data to obtain an encoding result, and perform a similarity comparison between the encoding result and the attribute information of the network node in the preset neural network model to obtain a comparison result, and determine a target network node based on the comparison result, wherein the attribute information of the target network node includes the first pose data; an output unit, configured to output the first posture data; The query unit is further configured to: construct a graph neural network; obtain historical visual data; determine key frame data in the training data and second pose data corresponding to the key frame based on the historical visual data; train the graph neural network based on the second pose data and the key frame data to obtain the preset neural network model; The key frame data includes target key frame data and remaining key frame data, the target key frame data is any key frame in the key frame data, and the remaining key frame data is the key frame data in the key frame data except the target key frame data.

11. A posture recovery device, characterized in that: include: A controller and a memory, wherein the memory stores a program or instruction, and the controller implements the steps of the method according to any one of claims 1 to 9 when executing the program or instruction in the memory.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A robot, characterized in that: include: The posture recovery device according to claim 10 or 11; or The readable storage medium of claim 12.

14. The robot according to claim 13, characterized in that The robot includes a sweeping robot.

Images