Method and system for determining and correcting course during navigation of shopping mall service robot

Abstract

The invention discloses a course determination and correction method during navigation of a shopping mall service robot, which comprises the following steps: when the shopping mall service robot cannot determine the course, acquiring an image in front of the current course, and identifying feature points in the image by using a trained target identification model; performing feature point comparison on the identified image and an image in a pre-established comparison database to obtain a closest database image; according to the feature point difference between the recognized image and the database image obtained through comparison, the recognized image is transformed to be consistent with the database image in combination with epipolar geometry, the angle difference and the position difference in the transformation process are calculated, and the current course of the shopping mall service robot is reversely deduced; and correcting the current course according to the angle difference between the current course of the shopping mall service robot and the destination orientation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of shopping mall service robots, in particular to the field of shopping mall service robot navigation heading determination and correction, and specifically to a shopping mall service robot navigation heading determination and correction method and system. BACKGROUND

[0002] With the rapid development of the field of intelligent robots, different types of robots have been developed one after another, among which service robots have also been concerned in recent years. In this type of robots, shopping mall service robots have developed rapidly and have also been put into use, but most of them are still in the early stage and have not reached complete intelligence in all aspects. At present, the research on shopping mall service robots at home and abroad is mostly target detection and recognition, such as establishing a target detection system of Faster R-CNN target detection model and a generalization zero sample learning model. There are also researches on the guide function of this type of robots, that is, the research on the navigation of shopping mall service robots, such as using D*Lite algorithm to realize the navigation function of shopping mall service robots. However, there are few researches in this field, and it is difficult to accurately determine the heading of the navigation in such a complex environment as a shopping mall, which may cause errors when guiding customers. SUMMARY

[0003] In order to overcome the shortcomings of the prior art, the present application provides a shopping mall service robot navigation heading determination and correction method, which can determine and correct the heading of the shopping mall service robot in real time during navigation based on visual SLAM, achieve more accurate navigation, and also realize the function of avoiding obstacles.

[0004] According to one aspect of the present application, a shopping mall service robot navigation heading determination and correction method is provided, comprising: When the shopping mall service robot cannot determine the heading, an image in front of the current heading is obtained, and a feature point in the image is recognized by using a trained target recognition model; The recognized image is compared with the image in the pre-established comparison database to obtain the closest database image; According to the feature point difference between the recognized image and the comparison database image, the recognized image is transformed to be consistent with the database image by combining epipolar geometry, and the angle difference and position difference in the transformation process are calculated to inversely deduce the current heading of the shopping mall service robot; The current heading is corrected according to the angle difference between the current heading of the shopping mall service robot and the destination direction.

[0005] As a further technical solution, the method further comprises: A shooting device is implanted in a shopping mall service robot, and a trained target recognition model is implanted in the shooting device to shoot and recognize environment images during navigation.

[0006] As a further technical solution, the training of the target recognition model further comprises: An image dataset in a shopping mall environment is obtained, and a target object in the image dataset is framed and marked; According to the image dataset, a YOLO algorithm is used for training to obtain a trained target recognition model.

[0007] As a further technical solution, the method further comprises: The trained target recognition model is implanted in the shooting device; The shopping mall environment is shot by the shooting device, and during the shooting process, visual SLAM is used and a dynamic rejection method is introduced to extract feature points of static objects, and the obtained video forms a comparison database.

[0008] As a further technical solution, the identified image is compared with the images in the pre-established comparison database in terms of feature points, comprising: The comparison database is extracted frame by frame, and each frame of image is compared with the currently shot image in terms of feature points to obtain a database image closest to the shot image.

[0009] As a further technical solution, the identified image is transformed to be consistent with the database image by combining epipolar geometry, and the angle difference and position difference in the transformation process are calculated to back-propagate the current heading of the shopping mall service robot, comprising: The epipolar constraint equation is derived; Based on the epipolar constraint equation, the essential matrix is solved; The pose of the shooting device during shooting is solved to obtain multiple possible solutions; One of the matching points is brought into the multiple possible solutions to determine a unique solution as the current heading.

[0010] As a further technical solution, the method further comprises: When navigating, the shopping mall service robot shoots the current environment image in real time through the implanted shooting device, uses the target recognition model for target recognition, and when an obstacle is recognized, re-plans the route.

[0011] According to an aspect of the present application, a system for determining and correcting the heading of a shopping mall service robot during navigation is provided, comprising a shopping mall service robot, and a photographing device and a processing device embedded in the shopping mall service robot; the photographing device is embedded with a trained target recognition model, and is used for photographing and recognizing environment images during navigation; the processing device compares an image of the front of the current heading of the shopping mall service robot when the heading cannot be determined with images in a pre-established comparison database, to obtain the closest database image; according to the feature point difference between the photographed image and the database image obtained by comparison, and in combination with epipolar geometry, the photographed image is transformed to be consistent with the database image, and the angle difference and position difference in the transformation process are calculated and the current heading of the shopping mall service robot is back calculated; and the current heading is corrected according to the angle difference between the current heading of the shopping mall service robot and the destination direction.

[0012] According to an aspect of the present application, a system for determining and correcting the heading of a shopping mall service robot during navigation is provided, comprising: a first main module, used for obtaining an image of the front of the current heading when the heading of the shopping mall service robot cannot be determined, and recognizing feature points in the image by using a trained target recognition model; a second main module, used for comparing the recognized image with images in a pre-established comparison database in terms of feature points, to obtain the closest database image; a third main module, used for transforming the recognized image to be consistent with the database image according to the feature point difference between the recognized image and the database image obtained by comparison, in combination with epipolar geometry, calculating the angle difference and position difference in the transformation process, and back calculating the current heading of the shopping mall service robot; a fourth main module, used for correcting the current heading according to the angle difference between the current heading of the shopping mall service robot and the destination direction.

[0013] According to an aspect of the present application, a non-transitory computer readable storage medium is provided, which stores computer instructions, and the computer instructions make the computer execute the method for determining and correcting the heading of a shopping mall service robot during navigation.

[0014] Compared with the prior art, the present application has the following beneficial effects: The present application determines and corrects the heading of a shopping mall service robot during navigation based on visual SLAM, and can improve the navigation accuracy of the shopping mall service robot during guidance, and can more efficiently complete the guidance work even in a complex environment such as a shopping mall, bring good service experience to customers, increase the customer flow of the shopping mall, and bring good economic benefits.

[0015] The technical hardware cost required by the application is also economical, without high cost investment, thereby reducing the burden, and meanwhile, the technical scheme provided by the application has good divergence, and can bring inspiration to other related fields. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0017] Figure 1 The method flowchart provided by the embodiment of the present application.

[0018] Figure 2 The YOLO algorithm training model flowchart provided by the embodiment of the present application.

[0019] Figure 3 The calculation flowchart of camera pose calculation by epipolar geometry provided by the embodiment of the present application. DETAILED DESCRIPTION

[0020] The present application determines the heading of the shopping mall service robot when guiding the customer and corrects the heading when the current heading is inconsistent with the target direction on the basis of visual SLAM. In the case of being able to determine the current heading and correct the heading, the shopping mall service robot can lead the customer to find the destination more efficiently in such a complex environment as a shopping mall, give the customer a good service experience, and at the same time, bring higher customer flow to the shopping mall to a certain extent.

[0021] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application. In addition, the technical features in each embodiment or single embodiment provided by the present application can be combined with each other to form new technical scheme, and this combination is not restricted by the order of steps and / or structure mode, but should be based on the realization by those skilled in the art, when the combination of technical schemes appears contradictory or cannot be realized, it should be considered that the combination of technical schemes does not exist, and is not within the protection scope required by the present application.

[0022] The embodiment of the present application provides a kind of based on visual SLAM's shopping mall service robot navigation time heading determination and correction method, through the training and implantation of target identification model under shopping mall environment, establish corresponding comparison database, the comparison database is usually the video stream of service environment.When shopping mall service robot guides work, shoot an image in the direction of current heading, compare the image with each frame image in comparison database, find out the most close image, determine the pose of current camera shooting by using epipolar geometry knowledge, to determine the current heading, again by the angle difference between destination direction and current heading, correct the current heading, overall process as shown in Figure 1 .

[0023] Model training as shown in Figure 2 , first, obtain the data set of model, generally static image data set;Subsequently, image data set is divided into two parts, one part is used for training, one part is used for inspection;Then, mark is framed in target in training image set;Then, training code is used to train training image set, and test image set is used to test model training result;Finally, judge whether the model precision meets the requirement, and end the training process when the precision meets the requirement.

[0024] S1, model training: the embodiment of the present application uses YOLO (You Only Look Once) algorithm to train the target identification model of the shopping mall served by robot, such as the sign in shopping mall, landmark building, trademark plate of fixed merchant and some static obstacles, the data source (photo) of each type of object mentioned above is put into training code, the corresponding model can be trained, and various human postures such as standing, walking, squatting, sitting and the like are trained in the same way.

[0025] S2, model implantation: the trained target identification model is implanted into oak binocular camera, here, oak binocular camera is used as technical hardware, oak binocular camera has better compatibility with yolo model, and can play a greater effect.

[0026] S3, the establishment of comparison database: comparison database is video containing shopping mall environment, with each frame image as a unit.The video of the shopping mall environment is shot by using OAK binocular camera, in the process of shooting, try to shoot the shopping mall in all directions as much as possible and try to record as many special buildings and eye-catching signs as possible, such as some trademarks in shopping mall, landmark buildings and the like.At the same time, use visual SLAM when shooting, add dynamic rejection mode, only sample the feature points of static objects, dynamic objects such as people are not sampled, so the video obtained is the comparison database.

[0027] S4, Object recognition and avoidance: Oak dual camera is implanted into a shopping mall service robot (other cameras can also be used as long as the effect is achieved), when it is guiding, it takes a picture of the current environment through the oak dual camera, and the oak dual camera uses the trained target recognition model to recognize, such as recognizing signs, trademarks, special buildings, obstacles, etc., when some obstacles are recognized or humans are recognized in front, a new route is planned to avoid.

[0028] S5, position, heading determination and heading correction: when the shopping mall service robot goes to a more open environment, such as the first floor hall of the mall, it cannot determine its own direction to a specific direction, at this time, the oak dual camera is used to take a picture of the front in its current heading, and the picture is compared with the pre-established comparison database. The specific method is to extract one frame by one frame from the comparison database, compare each frame of image with the current picture, and get a database image closest to the captured image by comparing the feature points in the image. Then use the knowledge of epipolar geometry, according to the difference of the angle and other information of the feature points in the two images, make the image change such as translation, rotation, etc., so that it changes to be consistent with the extracted image in the comparison database, calculate the angle difference and position difference of the image change, and deduce the specific direction of the current shopping mall service robot, and then calculate the angle difference between the current heading and the direction of the destination, and correct the current heading, so as to realize the determination and correction of the heading, and in the process of image comparison, the current position is also determined.

[0029] Regarding the determination and correction of epipolar geometry, as shown in Figure 3 , the specific method is as follows: First, in the image closest to the current captured image extracted from the comparison database, there are:

[0030] Among them, is the pixel coordinates of the feature points in the image, which is known; is the depth between the camera and the feature point, which is unknown; is the intrinsic matrix of the camera, which is known by default, ; is the coordinates of the feature points in the camera coordinate system, which is unknown.

[0031] Similarly, in the current captured image, there are also:

[0032] is the pixel coordinates of the feature points in the current captured image, is the coordinate of the current image in the camera coordinate system.

[0033] When the camera pose transformation is solved, the current heading is solved, and the camera pose transformation can be expressed as

[0034] 、 is the rotation matrix and translation vector when the camera coordinate system moves between the front and back.

[0035] Substitute equations (2) and (3) into equation (4), we have

[0036] Left multiply both sides by , we have

[0037] Equation (6) is the epipolar constraint equation, and the equation is solved as follows: First, let

[0038] where E is called the essential matrix, and F is called the fundamental matrix, because there are two matched feature points , , we have

[0039] Cross multiply the left side , we have

[0040] Dot product of the left side , we have

[0041] Therefore, we have

[0042] Considering the scale equivalence of the essential matrix, E has eight degrees of freedom, and eight pairs of matching points are needed to calculate E. After obtaining E by solving equation (12) through SVD, the pose 、 is solved, and E is singular value decomposed:

[0043] where 、 is an orthogonal matrix, is a singular value matrix.

[0044] Let

[0045] have

[0046] wherein

[0047] so

[0048] since , All can satisfy the singular value matrix of the essential matrix for The form of E, so that the decomposition of E has two forms; and by underdetermined equation E, because of the lack of scale constraints, can get E, -E two cases, finally will appear four groups of possible , The solution, when only one of the matching points into four possible solutions, get two camera coordinates when the depth is positive, indicating that the solution is correct, also the camera pose transformation is solved, that is, the current heading, and then the correction to the target direction.

[0049] The preliminary effect that can be achieved: can successfully complete the training of the model and the establishment of the comparison database, on this basis, basically realize the determination and correction of the heading of the shopping mall service robot when guiding customers, realize relatively accurate navigation, bring good service experience to customers, increase the customer flow of the shopping mall, and bring good economic benefits.

[0050] The implementation basis of each embodiment of the present application is that the processing of the device with processor function is programmed. Therefore, in engineering practice, the technical scheme and function of each embodiment of the present application are packaged into various modules. Based on this actual situation, on the basis of each embodiment described above, the embodiment of the present application provides a system for determining and correcting the heading of a shopping mall service robot during navigation, which is used to execute the method for determining and correcting the heading of the shopping mall service robot during navigation in the method embodiment.

[0051] The system comprises: a first main module configured to acquire an image in front of a current heading of the shopping mall service robot when the shopping mall service robot cannot determine the heading, and recognize feature points in the image by using a trained target recognition model; a second main module configured to compare the recognized image with images in a pre-established comparison database to obtain a closest database image; a third main module configured to transform the recognized image to be consistent with the database image according to a feature point difference between the recognized image and the database image obtained by comparison, in combination with epipolar geometry, calculate an angle difference and a position difference in the transformation process, and inversely deduce the current heading of the shopping mall service robot; and a fourth main module configured to correct the current heading according to an angle difference between the current heading of the shopping mall service robot and a destination direction.

[0052] The system for determining and correcting the heading of a shopping mall service robot in navigation provided by the embodiment of the application is based on the current situation of determining and correcting the navigation heading in a shopping mall environment, uses the foregoing modules, and can determine and correct the heading of the shopping mall service robot in real time during navigation based on visual SLAM, so as to achieve relatively accurate navigation and also achieve the function of avoiding obstacles.

[0053] It should be noted that the system embodiments provided by the application are used to implement the methods in the method embodiments and are also used to implement the methods in other method embodiments provided by the application. The difference is only that the corresponding functional modules are set, the principle is basically the same as that of the foregoing system embodiments provided by the application, as long as the person skilled in the art improves the modules in the foregoing system embodiments by combining the technical features in other method embodiments to obtain the corresponding technical means and the technical solutions formed by the technical means, and the technical solutions are ensured to have practicality, and then the corresponding system embodiments are obtained to implement the methods in other method embodiments.

[0054] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a system for determining and correcting the heading of a shopping mall service robot during navigation, comprising a shopping mall service robot, and a photographing device and a processing device implanted in the shopping mall service robot; the photographing device has a trained target recognition model implanted therein, for photographing and recognizing the environment image during navigation; the processing device compares the image of the front of the current heading of the shopping mall service robot when the heading cannot be determined with the images in the pre-established comparison database, to obtain the closest database image; according to the feature point difference between the photographed image and the database image obtained by comparison, combined with epipolar geometry, the photographed image is transformed to be consistent with the database image, the angle difference and position difference in the transformation process are calculated, and the current heading of the shopping mall service robot is back calculated; and the current heading is corrected according to the angle difference between the current heading of the shopping mall service robot and the destination direction.

[0055] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a non-transitory computer readable storage medium storing computer instructions, which cause the computer to perform the method for determining and correcting the heading of a shopping mall service robot during navigation as follows: When the heading of the shopping mall service robot cannot be determined, an image of the front of the current heading is obtained, and a trained target recognition model is used to recognize the feature points in the image; The recognized image is compared with the images in the pre-established comparison database in terms of feature points, to obtain the closest database image; According to the feature point difference between the recognized image and the database image obtained by comparison, combined with epipolar geometry, the recognized image is transformed to be consistent with the database image, the angle difference and position difference in the transformation process are calculated, and the current heading of the shopping mall service robot is back calculated; The current heading is corrected according to the angle difference between the current heading of the shopping mall service robot and the destination direction.

[0056] In summary of the embodiments, the present application realizes the accurate positioning of the shopping mall service robot and the correction of the fine heading of the shopping mall service robot during the guidance of the customers, so that the shopping mall service robot can also advance to the correct target direction in a complex environment. Moreover, the present application attracts customers to a certain extent, improves the customer volume of the shopping mall, improves the turnover, and the cost of visual SLAM is relatively low, which is suitable for consumer-grade products and lightweight applications.

[0057] It should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A method for heading determination and correction during navigation of a mall service robot, characterized by, The method comprises the following steps: When the shopping mall service robot cannot determine the heading direction, an image in front of the current heading direction is acquired, and a feature point in the image is recognized by using a trained target recognition model; The recognized image is compared with images in a pre-established comparison database in terms of feature points, and the closest database image is obtained; According to the difference between the recognized image and the comparison database image in terms of feature points, and in combination with epipolar geometry, the recognized image is transformed to be consistent with the database image, and the angle difference and position difference in the transformation process are calculated to inversely deduce the current heading direction of the shopping mall service robot; According to the angle difference between the current heading direction of the shopping mall service robot and the destination direction, the current heading direction is corrected.

2. The method for determining and correcting a heading of a shopping mall service robot in navigation according to claim 1, wherein, The method further comprises the following steps: A shooting device is implanted in the shopping mall service robot, and the shooting device is implanted with a trained target recognition model, which is used for shooting and recognizing environment images during navigation.

3. The method for navigation course determination and correction of a shopping mall service robot according to claim 2, wherein, The training of the target recognition model further comprises the following steps: Image data sets in a shopping mall environment are acquired, and a target object in the image data sets is framed and marked; According to the image data sets, a YOLO algorithm is used for training to obtain a trained target recognition model.

4. The method for navigation course determination and correction of a shopping mall service robot according to claim 3, wherein, The method further comprises the following steps: The trained target recognition model is implanted in the shooting device; The shopping mall environment is shot by using the shooting device, and in the shooting process, a visual SLAM is used and a dynamic elimination method is introduced to extract feature points of static objects, and a comparison database is formed by using the obtained video.

5. The method for navigation course determination and correction of a shopping service robot according to claim 1, wherein, The recognized image is compared with images in a pre-established comparison database in terms of feature points, and the closest database image is obtained. The comparison database is frame-by-frame separated, and each frame of image is compared with the currently shot image in terms of feature points to obtain a database image closest to the shot image.

6. The method for navigation course determination and correction of a shopping service robot according to claim 1, wherein, In combination with epipolar geometry, the recognized image is transformed to be consistent with the database image, and the angle difference and position difference in the transformation process are calculated to inversely deduce the current heading direction of the shopping mall service robot, which comprises the following steps: An epipolar constraint equation is derived; Based on the epipolar constraint equation, an essential matrix is solved; A pose of the shooting device during shooting is solved to obtain multiple possible solutions; One matching point is brought into the multiple possible solutions to determine a unique solution as the current heading direction.

7. The method for navigation course determination and correction of a shopping service robot according to claim 2, wherein, The method further comprises the following steps: During navigation, the shopping mall service robot shoots a current environment image in real time by using the implanted shooting device, and target recognition is performed by using the target recognition model, and when an obstacle is recognized, route planning is performed again.

8. A system for heading determination and correction for a shopping mall service robot navigation, characterized by, The mall service robot comprises a shooting device and a processing device implanted in the mall service robot; the trained target recognition model is implanted in the shooting device, and is used for shooting and recognizing environment images during navigation; the processing device compares the image in front of the current heading direction when the mall service robot cannot determine the heading direction with the images in the pre-established comparison database, and obtains the closest database image; according to the feature point difference between the shot image and the comparison database image, the shot image is transformed to be consistent with the database image by combining epipolar geometry, the angle difference and position difference in the transformation process are calculated, and the current heading direction of the mall service robot is back calculated; and the current heading direction is corrected according to the angle difference between the current heading direction of the mall service robot and the destination direction.

9. A system for heading determination and correction for a shopping mall service robot navigation, characterized by, The mall service robot comprises: a first main module, which is used for acquiring the image in front of the current heading direction when the mall service robot cannot determine the heading direction, and recognizing the feature points in the image by using the trained target recognition model; a second main module, which is used for comparing the recognized image with the images in the pre-established comparison database, and obtaining the closest database image; a third main module, which is used for transforming the recognized image to be consistent with the database image according to the feature point difference between the recognized image and the comparison database image, combining epipolar geometry, calculating the angle difference and position difference in the transformation process, and back calculating the current heading direction of the mall service robot; a fourth main module, which is used for correcting the current heading direction according to the angle difference between the current heading direction of the mall service robot and the destination direction.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions make the computer execute the method for determining and correcting the heading direction of the mall service robot during navigation according to any one of claims 1-7.

Images