A fast panoramic vision navigation method for indoor mobile robots based on twin neural networks

Abstract

The present invention discloses a method for rapid panoramic vision navigation of an indoor mobile robot based on a twin neural network, which comprises: using a mobile robot equipped with a panoramic imaging device to collect a panoramic image of an indoor environment as original panoramic image data; performing coordinate value naming processing on different image data according to the shooting position information of the original panoramic image data, and matching any two panoramic images with coordinate value naming processing to obtain a panoramic image pair; determining training set data for network model training and test set data for network model testing; constructing a twin neural network model for panoramic vision navigation of an indoor mobile robot; inputting the matched panoramic image pair into the twin neural network model, and after training and testing the network model, running the network model to control the mobile robot to perform autonomous navigation. The present invention can effectively realize rapid and accurate autonomous navigation of a mobile robot in an indoor environment.

Description

Technical Field

[0001] The present invention relates to the field of robot navigation, and in particular to a panoramic vision rapid navigation method for an indoor mobile robot based on a twin neural network. Background Art

[0002] Currently, panoramic vision navigation technology is a hot research direction in the field of mobile robot navigation. With the rapid development of artificial intelligence technology, the bottlenecks exposed by traditional panoramic vision navigation technology, such as low information extraction efficiency, inaccurate direction calculation accuracy, and poor navigation effect, have been broken through. Compared with traditional image processing technology, deep learning technology builds deep neural networks to learn the mapping relationship between input and output to achieve accurate and fast autonomous navigation of the robot. However, the current mainstream deep learning technology generally requires a large amount of data support, and the existing indoor mobile robot navigation methods generally require complex autonomous positioning and map construction processes.

[0003] Existing deep learning-based indoor mobile robot navigation generally relies on simultaneous positioning and map building technology, which requires the robot system to have strong real-time data processing and storage capabilities, especially the processing of complex environmental maps and robot location information. The deep neural network model needs to rely on a large amount of data training to demonstrate ideal navigation decision-making capabilities. This navigation method requires the robot to bear a huge computing load, which in turn causes a significant increase in the manufacturing and maintenance costs of the robot. On the contrary, when the manufacturing and maintenance costs of the robot are limited, the above navigation method will not be able to meet the requirements for navigation accuracy and real-time performance, which will affect the autonomous navigation effect of the mobile robot. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, for the autonomous navigation of indoor mobile robots, and to help realize the accurate and fast navigation of indoor mobile robots, the present invention proposes a systematic and reliable panoramic vision rapid navigation method for indoor mobile robots based on twin neural networks.

[0005] To this end, a first object of an embodiment of the present invention is to provide a panoramic vision rapid navigation method for an indoor mobile robot based on a twin neural network, which comprises the following steps:

[0006] 1) Using a mobile robot equipped with a panoramic imaging device to collect panoramic images of the indoor environment as original panoramic image data;

[0007] 2) performing coordinate value naming processing on different image data according to the shooting location information of the original panoramic image data, and matching any two panoramic images with the coordinate value naming processing to obtain a panoramic image pair;

[0008] 3) Determine the training set data for network model training and the test set data for network model testing;

[0009] 4) Construct a twin neural network model for rapid navigation of indoor mobile robots with panoramic vision;

[0010] 5) The matched panoramic image pair is input into the twin neural network model. After training and testing the network model, the network model is run to control the mobile robot for autonomous navigation.

[0011] Preferably, the step of collecting a panoramic image of an indoor environment comprises the following steps:

[0012] 1) Select an indoor rectangular area for the robot to move, and evenly divide the area into multiple square grids with a side length of k centimeters;

[0013] 2) Setting the intersection of the square grid as the location where the panoramic image is to be collected;

[0014] 3) Controlling a mobile robot equipped with a panoramic imaging device to capture a panoramic image at a location to be captured;

[0015] Preferably, the method of performing coordinate value naming processing on different image data according to shooting location information of the original panoramic image data, and matching any two panoramic images with coordinate value naming processing to obtain a panoramic image pair includes the following steps:

[0016] 1) Set the upper left corner of the acquisition position as the origin to establish a coordinate system, thereby determining the coordinates of all acquisition positions. The coordinate value is the name of the corresponding panoramic image;

[0017] 2) For all panoramic images with coordinate value naming processing, randomly select two panoramic images and calculate the corresponding direction vector according to the coordinate values ​​of the two images;

[0018] 3) setting the calculated direction vector mark as a label of the corresponding two panoramic images, and saving the two panoramic images with the label in a unified folder, which is defined as a group of panoramic image pairs;

[0019] Preferably, the determination of the training set data for network model training and the test set data for network model testing specifically refers to stipulating that a portion of panoramic image pairs are used for subsequent network model training, i.e., training set data; and another portion of image pairs are used for subsequent network model testing, i.e., test set data.

[0020] Preferably, the construction of a twin neural network model for panoramic vision rapid navigation of an indoor mobile robot comprises the following steps:

[0021] 1) Designing a twin neural network architecture

[0022] A twin sub-network with the residual network ResNet-50 as the basic module was determined, and based on the output of the twin sub-network, a fine-tuning sub-network with a total of 10 convolutional layers was further designed, ultimately realizing the design of a twin neural network architecture for panoramic vision rapid navigation of indoor mobile robots;

[0023] 2) Determine the input and output of the network model

[0024] According to the designed twin neural network architecture, it is preliminarily determined that its input is two images in a set of panoramic image pairs, and its output is the direction vector label corresponding to the two images;

[0025] 3) Determine the loss function of the network model

[0026] Select the normalized exponential function as the loss function of the designed neural network architecture;

[0027] Preferably, the matched panoramic image pair is input into the twin neural network model, and after the network model is trained and tested, the network model is run to control the mobile robot to perform autonomous navigation, including the following steps:

[0028] 1) Training network model

[0029] After the network model is determined, the training set data is input into the network model. The network model explores the mapping relationship between the two panoramic images and their corresponding labels based on the input image data and corresponding labels in the training set, and preliminarily learns the ability to judge the navigation direction.

[0030] Use all training set data to train the network model, compare the actual output of the network model with the label, and observe the changes in the normalized exponential function curve until the curve converges;

[0031] 2) Test network model

[0032] After the network model is trained, the test set data is input into the network model; for the difference between the actual output of the network model and the corresponding label, it is quantitatively determined whether the network model has the ability to judge the navigation direction: if it has, it proceeds to the step of running the network model; if not, it returns to the step of obtaining the panoramic image pair to reallocate the training set and the test set;

[0033] 3) Run the network model

[0034] After the robot pre-shoots and saves a panoramic image of the target position as one of the inputs in the network model, a panoramic image of the robot at the current position is shot as another input in the network model, and the network model is run to control the robot to continuously move along the direction vector output by the network model.

[0035] The panoramic vision rapid navigation method for an indoor mobile robot based on a twin neural network of the present invention can effectively realize accurate and rapid autonomous navigation of an indoor mobile robot equipped with a panoramic imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. It should be noted that the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0037] Figure 1 It is a schematic diagram of the process flow of a method for rapid panoramic vision navigation of an indoor mobile robot according to an embodiment of the present invention;

[0038] Figure 2 A map of locations where panoramic image data is to be collected for panoramic vision rapid navigation of an indoor mobile robot according to an embodiment of the present invention;

[0039] Figure 3 This is a diagram of a twin neural network for panoramic vision rapid navigation of an indoor mobile robot according to an embodiment of the present invention;

[0040] Figure 4 The present invention is a flowchart of network model training, testing and operation of a method for rapid panoramic vision navigation of an indoor mobile robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] like Figure 1 As shown in FIG. 1 , a flowchart of a method for rapid panoramic vision navigation of an indoor mobile robot based on a twin neural network according to an embodiment of the present invention is shown. The specific steps are as follows:

[0043] S1: Use a mobile robot equipped with a panoramic imaging device to collect a panoramic image of an indoor environment as raw panoramic image data.

[0044] In this embodiment, the following method can be used to obtain the original panoramic image data: first, select an indoor rectangular area for the robot to move, and evenly divide the area into multiple square grids with a side length of k centimeters, then set the intersection of the square grids as the position to be collected for the panoramic image, and finally control the mobile robot equipped with a panoramic imaging device to take a panoramic image at the position to be collected.

[0045] In this embodiment, the selected indoor rectangular area is not strictly limited and can be an indoor home environment, laboratory environment, industrial workshop environment, etc.; the side length of the set square grid is k centimeters, and its value is related to the actual navigation needs. The larger the k value, the larger the interval between the panoramic image acquisition positions, and the less image data obtained in the same area, which can reduce the computing load of the mobile robot, but the corresponding navigation accuracy will be reduced; the smaller the k value, the smaller the interval between the panoramic image acquisition positions, and the more image data obtained in the same area, which helps to improve the navigation accuracy of the mobile robot, but the corresponding computing load will increase. If there is no specific requirement for navigation accuracy or computing load, the k value can be adjusted so that there are 10-15 panoramic image acquisition positions in each row or column of the rectangular area.

[0046] Figure 2 The figure shows the locations of panoramic image data to be collected for the panoramic vision rapid navigation of the indoor mobile robot adopted in this embodiment. The black solid dots in the figure are the intersection points of the square grid, that is, the locations of all panoramic images to be collected.

[0047] In this embodiment, the mobile robot equipped with the panoramic imaging device can be controlled by using the preset control instructions that come by default in the robot to control the robot to move autonomously to the position to be collected, or the mobile robot can be placed at the position to be collected by manually moving and placing it. This embodiment does not limit the specific orientation of the robot at any collection position, but it is necessary to ensure that the orientation of the robot should be the same from beginning to end. Under this premise, the panoramic imaging device installed on the mobile robot is used to capture a panoramic image at the current position. The purpose of this step is to control the panoramic images captured by the robot to be in the same direction, thereby reducing the pressure of the robot on image processing and feature extraction between different images, and reducing the robot's computational load.

[0048] S2: performing coordinate value naming processing on different image data according to the position information of the original panoramic image data, and matching any two panoramic images with the coordinate value naming processing to obtain a panoramic image pair.

[0049] In this embodiment, the acquisition position at the upper left corner is set as the origin to establish a coordinate system, thereby determining the coordinates of all acquisition positions, and the coordinate value is the name of the corresponding panoramic image. Figure 2The numbers marked in are the horizontal and vertical coordinate values ​​of the established coordinate system, and the position coordinates of each panoramic image are named after the coordinate values ​​of the image.

[0050] In this embodiment, for all panoramic images with coordinate value naming processing, two panoramic images are randomly selected, and the corresponding direction vectors are calculated according to the coordinate values ​​of the two images. The direction vector is presented in the form of angle. For example, assuming that the two selected panoramic images are image 1 and image 2, and their names are (x1, y1) and (x2, y2) respectively, then the corresponding direction vectors θ of the two images are 12 for:

[0051]

[0052] In this embodiment, the following method can be used to obtain the remote sensing image pair: set the calculated direction vector mark as the label of the corresponding two panoramic images, save the two panoramic images with the label in a unified folder, and the folder is defined as a group of panoramic image pairs. Using the above method, all two panoramic images with direction vector labels are combined into a panoramic image pair.

[0053] S3: Determine the training set data for network model training and the test set data for network model testing.

[0054] In this embodiment, all the panoramic image pairs obtained can be divided into training set data and test set data according to a certain quantity ratio. For example, according to the common division principles in the current artificial intelligence field, 70% of all panoramic image pairs can be randomly selected as the training set of the network model to train the network model's ability to autonomously judge navigation decisions; the other 30% can be used as the test set of the network model to test the actual decision-making effect of the model. It should be noted that the above division ratio is not fixed. In addition to 70%:30%, common division ratios can also be 80%:20% or 90%:10%, etc.

[0055] S4: Construct a twin neural network model for fast panoramic vision navigation of indoor mobile robots.

[0056] The purpose of this step is to achieve rapid panoramic vision navigation of indoor mobile robots by constructing a twin neural network model. The advantage of this embodiment is that the twin neural network can effectively extract the features with differences between the two input images, and then accurately judge the similarity between the two input images, and infer the moving direction of the mobile robot, thereby truly achieving the purpose of accurate and rapid autonomous navigation. The detailed process of this step can be found in reference [1].

[0057] The following are the specific steps for constructing a twin neural network model in this embodiment:

[0058] S41: Determine the twin sub-network with the residual network ResNet-50 as the basic module, and based on the output of the twin sub-network, further design a fine-tuning sub-network containing a total of 10 convolutional layers, and finally realize the design of a twin neural network architecture for panoramic vision rapid navigation of indoor mobile robots.

[0059] Figure 3 The figure shows an example of a twin neural network for panoramic vision rapid navigation of an indoor mobile robot according to an embodiment of the present invention. In this embodiment, the designed twin neural network architecture includes two sub-networks. The first sub-network is a twin sub-network, which is composed of two residual networks ResNet-50 with exactly the same structure, and can be used to extract features with differences between two input images. The second sub-network is a fine-tuning sub-network, which includes a total of 10 convolutional layers and is connected by a residual learning method. It can be used to further analyze and judge the differential features extracted in the twin sub-network and help obtain effective output. The detailed structure and process of the residual network ResNet-50, convolutional layer and residual learning method involved in this step can be found in reference [2].

[0060] S42: According to the designed twin neural network architecture, it is preliminarily determined that its input is two images in a set of panoramic image pairs, and its output is the direction vector labels corresponding to the two images.

[0061] The purpose of this step is to enable the designed network model to have the ability to use the input image to infer the output direction. In this embodiment, the input of the designed twin neural network architecture is specified as two images in a set of panoramic image pairs, and the differential features of the two images are extracted through the twin sub-network, and the corresponding direction vector is generated through the fine-tuning sub-network, thereby constructing a mapping relationship from the two panoramic image input data to the direction vector output data.

[0062] S43: Select a normalized exponential function as the loss function of the designed neural network architecture.

[0063] The purpose of this step is to constrain the actual output of the network model to be differentiated from the direction vector label representing the true value, and to reduce this difference so that the actual output of the network model can be close to the true value direction vector label. In this embodiment, a normalized exponential function is selected as the loss function, so as to infer the learning ability of the network model through the change of the loss function of the network model during the training phase. For a detailed introduction to the normalized exponential function involved in this step, please refer to reference [3].

[0064] In the following S5 step, the general ideas used in the current field of artificial intelligence for training, testing, and running network models are adopted. Figure 4 The figure shows a flow chart of network model training, testing and operation in this embodiment.

[0065] S5: Input the matched panoramic image pair into the twin neural network model. After training and testing the network model, run the network model to control the mobile robot to perform autonomous navigation.

[0066] In this embodiment, the training set and test set data determined in step S3 are input into the network model determined in step S4 for training and testing, thereby helping the network model to initially have the ability to control the robot to perform accurate and fast autonomous navigation. The detailed process in this step can be found in reference [1].

[0067] The specific steps include:

[0068] S51: Training network model

[0069] The network model determined in step S4 is trained using the training set data determined in step S3. During the training process, the normalized exponential function curve is observed. When the curve gradually converges to no longer violently oscillates, it can be preliminarily determined that the network model training is completed.

[0070] It should be noted that the above method is a general method for judging the network training status in the field of artificial intelligence. Under normal circumstances, any loss function, including the normalized exponential function, will experience a relatively obvious oscillation and decline process in the early stage of network model training. As the number of training times increases, the loss function curve tends to converge steadily, and the oscillation effect is significantly weakened. This state can indirectly indicate that the current network model has been trained. For details of the above description, please refer to references [1]-[2].

[0071] S52: Test network model

[0072] The network model determined in step S4 is tested using the test set data determined in step S3. For the difference between the actual output of the network model and the corresponding label, a quantitative judgment is made as to whether the network model has the ability to judge the navigation direction. The quantitative standard is specifically expressed as: the direction vector value of the actual output is subtracted from the label. When the difference is lower than the preset threshold, it is considered that the network model has the ability to judge the navigation direction. The setting principle of the threshold is related to the actual needs. Obviously, the setting of the threshold will affect the final effect of the network model. The lower the threshold, the closer the actual output of the network model is to the true value label, the stronger the network's ability to judge the navigation direction, but the higher the difficulty of training the network model; conversely, the higher the threshold, the greater the difference between the actual output of the network model and the true value label, the lower the difficulty of training the network model, but the weaker the network's ability to judge the navigation direction.

[0073] It should be noted that if the quantitative judgment network model already has the ability to judge the navigation direction, the next step of running the network model can be entered; but if the quantitative judgment network model already has the ability to judge the navigation direction, return to step S3 to reallocate the training set and the test set.

[0074] S53: Run the network model

[0075] After completing the training and testing of the network model using steps S52 and S53, it is determined that the current network model has the ability to control the panoramic vision rapid navigation of the indoor mobile robot. At this time, the network model can be run to truly realize the panoramic vision rapid navigation of the indoor mobile robot, which is specifically manifested as follows: first, control the mobile robot to shoot and save a panoramic image at the target position in advance, and fix it as one of the inputs in the network model; second, place the mobile robot in the original position to be set off, shoot a panoramic image of the robot at this position, and use it as another input of the network model; finally, run the network model, the model automatically outputs a direction vector (i.e., navigation direction), and the mobile robot moves to the target position according to the navigation direction until it reaches the target position, ending this autonomous navigation process.

Claims

1. A panoramic vision rapid navigation method for indoor mobile robots based on twin neural networks, characterized in that: The steps include: 1) Using a mobile robot equipped with panoramic imaging equipment to collect panoramic images of the indoor environment as raw panoramic image data; 2) According to the shooting location information of the original panoramic image data, coordinate value naming processing is performed on different image data, and any two panoramic images with coordinate value naming processing are matched to obtain a panoramic image pair; 3) Determine the training set data for network model training and the test set data for network model testing; 4) Construct a twin neural network model for rapid navigation of indoor mobile robots with panoramic vision; 5) Input the matched panoramic image pair into the twin neural network model. After training and testing the network model, run the network model to control the mobile robot to perform autonomous navigation; The construction of a twin neural network model for panoramic vision rapid navigation of an indoor mobile robot includes the following steps: 1) Designing a twin neural network architecture A twin sub-network with the residual network ResNet-50 as the basic module was determined, and based on the output of the twin sub-network, a fine-tuning sub-network with a total of 10 convolutional layers was further designed, ultimately realizing the design of a twin neural network architecture for panoramic vision rapid navigation of indoor mobile robots; 2) Determine the input and output of the network model According to the designed twin neural network architecture, it is preliminarily determined that its input is two images in a set of panoramic image pairs, and its output is the direction vector label corresponding to the two images; 3) Determine the loss function of the network model The normalized exponential function is selected as the loss function of the designed neural network architecture.

2. A method for rapid indoor mobile robot panoramic vision navigation based on a twin neural network as claimed in claim 1, characterized in that: The method of collecting the panoramic image of the indoor environment comprises the following steps: 1) Select an indoor rectangular area for the robot to move, and evenly divide the area into multiple square grids with a side length of k; 2) Set the intersection of the square grid as the location where the panoramic image is to be collected; 3) Control a mobile robot equipped with a panoramic imaging device to capture a panoramic image at the location to be captured.

3. A method for rapid indoor mobile robot panoramic vision navigation based on a twin neural network as claimed in claim 1, characterized in that: The method of performing coordinate value naming processing on different image data according to the shooting location information of the original panoramic image data, and matching any two panoramic images with the marking processing to obtain a panoramic image pair includes the following steps: 1) Set the upper left corner of the acquisition position as the origin to establish the coordinate axis, so as to determine the coordinates of all acquisition positions. The coordinate value is the name of the corresponding panoramic image; 2) For all panoramic images with coordinate value naming processing, randomly select two panoramic images and calculate the corresponding direction vector according to the coordinate values ​​of the two images; 3) Setting the calculated direction vector tag as a label of the corresponding two panoramic images, and saving the two panoramic images with the label in a unified folder, which is defined as a group of panoramic image pairs.

4. The method for rapid indoor mobile robot panoramic vision navigation based on a twin neural network as claimed in claim 1, characterized in that: The determination of the training set data for network model training and the test set data for network model testing specifically refers to stipulating that a portion of panoramic image pairs are used for subsequent network model training, i.e., training set data; and another portion of image pairs are used for subsequent network model testing, i.e., test set data.

5. A method for rapid indoor mobile robot panoramic vision navigation based on a twin neural network as claimed in claim 1, characterized in that: The matched panoramic image pair is input into the twin neural network model, and after the network model is trained and tested, the network model is run to control the mobile robot to perform autonomous navigation, including the following steps: 1) Training network model After the network model is determined, the training set data is input into the network model. The network model explores the mapping relationship between the two panoramic images and their corresponding labels based on the input image data and corresponding labels in the training set, and preliminarily learns the ability to judge the navigation direction. Use all training set data to train the network model, compare the actual output of the network model with the label, and observe the changes in the normalized exponential function curve until the curve converges; 2) Test network model After the network model is trained, the test set data is input into the network model; for the difference between the actual output of the network model and the corresponding label, it is quantitatively determined whether the network model has the ability to judge the navigation direction: if it has, it proceeds to the step of running the network model; if not, it returns to the step of obtaining the panoramic image pair to reallocate the training set and the test set; 3) Run the network model After the robot pre-shoots and saves a panoramic image of the target position as one of the inputs in the network model, a panoramic image of the robot at the current position is shot as another input in the network model, and the network model is run to control the robot to continuously move along the direction vector output by the network model.

Images