Method and apparatus for map building and visual positioning of a robot

Abstract

The application provides a robot map construction and visual positioning method and device, wherein the robot map construction method comprises the following steps: acquiring an image frame sequence through a camera, then performing two-dimensional code feature detection and visual feature point extraction on the image frame sequence, performing pose calculation on the two-dimensional code features, and performing triangulation depth estimation on the visual feature points in combination with the mileage information, so as to obtain the spatial position information of the two-dimensional code features and the spatial position information of the visual feature points in the image frame sequence. Then, a map in the robot visual environment is constructed according to the spatial position information of the visual feature points and the spatial position information of the two-dimensional code features. Based on the above method, the mileage information and the two-dimensional code information are fused to construct the map in the robot visual environment, so that the problem that the mobile robot is prone to positioning loss in scenes with many dynamic objects, low texture and poor lighting conditions can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and relates to a map construction and visual positioning method and device of a robot, and further relates to an electronic device and a storage medium for executing the map construction method of the robot and the visual positioning method of the robot. BACKGROUND

[0002] With the development of computer technology and artificial intelligence, intelligent autonomous mobile robots have become an important research direction and research hotspot in the field of robots. The positioning and map creation of mobile robots are hot research problems in the field of autonomous mobile robots. Robots usually construct a sparse landmark map containing geometric information based on a simultaneous localization and mapping (SLAM) technology, which can be used to perform a navigation positioning task. However, at present, mobile robots are prone to positioning loss in scenes with many dynamic objects, low texture, poor lighting conditions, and the like. Positioning loss will cause the mobile robot to fail to navigate and affect the task execution of the mobile robot. SUMMARY

[0003] Therefore, the embodiments of the present application provide a visual positioning method, device, and equipment of a robot and a storage medium, which can solve the problem that mobile robots are prone to positioning loss in scenes with many dynamic objects, low texture, poor lighting conditions, and the like.

[0004] A first aspect of the embodiments of the present application provides a map construction method of a robot, comprising: acquiring image information in a visual environment of the robot through a camera, the image information being represented as a sequence of image frames; performing two-dimensional code feature detection on the sequence of image frames, if a two-dimensional code exists in the sequence of image frames, performing pose calculation on the two-dimensional code feature to obtain spatial position information of the two-dimensional code feature; performing feature point extraction on each frame of the sequence of image frames to obtain visual feature points of each image in the sequence of image frames and generate a corresponding feature descriptor based on each visual feature point; performing adjacent frame feature point matching on the sequence of image frames according to the visual feature points and the feature descriptors corresponding to the visual feature points, if the visual feature points in two adjacent images are matched, acquiring mileage information when the two adjacent images are collected, and performing triangulation depth estimation on the visual feature points in the two adjacent images according to the mileage information to obtain spatial position information of the visual feature points; and constructing a map in the visual environment of the robot according to the spatial position information of the visual feature points and the spatial position information of the two-dimensional code feature.

[0005] In a possible implementation, before the step of performing two-dimensional code feature detection on the image frame sequence, if a two-dimensional code exists in the image frame sequence, performing pose calculation on the two-dimensional code feature to obtain spatial position information of the two-dimensional code feature, the step further includes: performing two-dimensional code feature detection on each image in the image frame sequence respectively, judging whether a two-dimensional code exists in the image, if a two-dimensional code exists, obtaining pixel coordinates of four corner points of the two-dimensional code feature; determining relative distances of the four corner points of the two-dimensional code feature according to the pixel coordinates of the four corner points of the two-dimensional code feature, performing pose estimation processing on the two-dimensional code feature according to the relative distances, and obtaining position information of the four corner points of the two-dimensional code feature in a camera coordinate system, and determining the position information of the four corner points of the two-dimensional code feature in the camera coordinate system as the spatial position information of the two-dimensional code feature.

[0006] In a possible implementation, before the step of constructing the map in the robot visual environment according to the spatial position information of the visual feature point and the spatial position information of the two-dimensional code feature, the step further includes: jointly processing multiple images in the image frame sequence, and performing optimization processing on the spatial position information of the visual feature point, the spatial position information of the two-dimensional code feature, and the pose information of the camera by using a bundle adjustment algorithm.

[0007] In a possible implementation, before the step of jointly processing multiple images in the image frame sequence, and performing optimization processing on the spatial position information of the visual feature point, the spatial position information of the two-dimensional code feature, and the pose information of the camera by using a bundle adjustment algorithm, the step further includes: setting weight values for the visual feature point and the two-dimensional code feature, wherein the weight value corresponding to the visual feature point is less than the weight value corresponding to the two-dimensional code feature.

[0008] A second aspect of the embodiment of the application provides a robot visual positioning method, which uses the map constructed by the robot map construction method in any one of the first aspect to perform visual positioning. The robot visual positioning method comprises: loading the map in the robot visual environment according to preset feature point type weight configuration information; collecting real-time images in a current visual environment of the robot by using the camera; performing two-dimensional code feature detection on the real-time images, if a two-dimensional code feature is detected in the real-time images, obtaining a target two-dimensional code identifier corresponding to the two-dimensional code feature; traversing the map in the robot visual environment according to the two-dimensional code identifier, if a two-dimensional code identifier matching the target two-dimensional code identifier is recorded in the map in the robot visual environment, performing visual positioning on the robot according to the two-dimensional code feature to obtain real-time position information of the robot.

[0009] In a possible implementation manner, after the step of performing the two-dimensional code feature detection on the real-time image, the method further includes: if no two-dimensional code feature is detected in the real-time image, performing feature point extraction processing on the real-time image to obtain a target visual feature point and generate a corresponding target feature point descriptor based on the target visual feature point; and traversing a map in a robot visual environment according to the target visual feature point and the target feature point descriptor corresponding to the target visual feature point, and if the map in the robot visual environment records a visual feature point and a target feature point descriptor corresponding to the visual feature point that correspond to the target visual feature point and the target feature point descriptor corresponding to the target visual feature point, performing visual positioning on the robot according to the target visual feature point and the target feature point descriptor corresponding to the target visual feature point to obtain real-time position information of the robot.

[0010] A third aspect of the embodiment of the present application provides a map construction device of a robot, including: an acquisition module configured to acquire image information in a robot visual environment through a camera, the image information being represented as a sequence of image frames; a calculation module configured to perform two-dimensional code feature detection on the sequence of image frames, and if a two-dimensional code exists in the sequence of image frames, performing pose calculation on the two-dimensional code feature to obtain spatial position information of the two-dimensional code feature; an extraction module configured to perform feature point extraction on the sequence of image frames frame by frame, to acquire visual feature points of each image in the sequence of image frames and generate a corresponding feature descriptor based on each visual feature point; an estimation module configured to perform adjacent frame feature point matching on the sequence of image frames according to the visual feature points and the feature descriptors corresponding to the visual feature points, and if visual feature points in two adjacent images are matched, acquiring mileage information when the two adjacent images are collected, and performing triangulation depth estimation on the visual feature points in the two adjacent images according to the mileage information to obtain spatial position information of the visual feature points; and a mapping module configured to construct a map in the robot visual environment according to the spatial position information of the visual feature points and the spatial position information of the two-dimensional code feature.

[0011] A fourth aspect of this application provides a visual positioning device for a robot, comprising: a loading module for loading a map of the robot's visual environment according to preset feature point type weight configuration information; a acquisition module for acquiring real-time images of the robot's current visual environment through a camera; a detection module for performing QR code feature detection on the real-time images, and if QR code features are detected in the real-time images, obtaining a target QR code identifier corresponding to the QR code features; and a positioning module for traversing the map of the robot's visual environment according to the QR code identifier, and if a QR code identifier matching the target QR code identifier is recorded in the map of the robot's visual environment, then visually positioning the robot according to the QR code features to obtain the robot's real-time position information.

[0012] A fifth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the electronic device, wherein the processor executes the computer program to implement the steps of the method provided in the first or second aspect.

[0013] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method provided in the first or second aspect.

[0014] The visual positioning method, apparatus, electronic device, and storage medium for robots provided in this application have the following beneficial effects:

[0015] This application acquires image frame sequences via a camera, then performs QR code feature detection and visual feature point extraction on the image frame sequences. By calculating the pose of the QR code features and combining odometry information to perform triangulation depth estimation of the visual feature points, the spatial location information of the QR code features and the spatial location information of the visual feature points in the image frame sequence are obtained. A map of the robot's visual environment is then constructed based on the spatial location information of the visual feature points and the QR code features. This method, by fusing odometry information and QR code information to construct a map of the robot's visual environment, solves the problem of localization loss that mobile robots easily experience in scenarios with many dynamic objects, low texture, and poor lighting conditions. Attached Figure Description

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0017] Figure 1 The implementation flowchart of the robot map construction method provided by the embodiment of the present application is shown in the following table.

[0018] Figure 2 The implementation flowchart of the method for acquiring the spatial position information of the two-dimensional code feature in the robot map construction method provided by the embodiment of the present application is shown in the following table.

[0019] Figure 3 The implementation flowchart of the robot visual positioning method provided by the embodiment of the present application is shown in the following table.

[0020] Figure 4 The implementation flowchart of the robot visual positioning method provided by the embodiment of the present application is shown in the following table.

[0021] Figure 5 The basic structure block diagram of the robot map construction device provided by the embodiment of the present application is shown in the following table.

[0022] Figure 6 The basic structure block diagram of the robot visual positioning device provided by the embodiment of the present application is shown in the following table.

[0023] Figure 7 The basic structure block diagram of the electronic device provided by the embodiment of the present application is shown in the following table.

[0024] Embodiments

[0025] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0026] Please refer to Figure 1 , Figure 1 The implementation flowchart of the robot map construction method provided by the embodiment of the present application is shown in the following table. Specifically, it can include the following steps S11 to S15.

[0027] S11: acquiring image information in the visual environment of the robot through a camera, the image information being represented as a sequence of image frames.

[0028] In this embodiment, the camera is mounted on the robot, and during the movement of the robot, the image information in the robot vision environment is obtained by video shooting or continuous image shooting at a fixed frequency through the camera. It can be understood that the image information obtained by video shooting or continuous image shooting at a fixed frequency is represented as a sequence of image frames, and the sequence of image frames contains a plurality of image frames, and each image frame has corresponding timestamp information. In a complex environment such as strong light change, the detection and recognition of markers such as two-dimensional codes in the visible light band have a great influence. In order to avoid this influence, in this embodiment, the selected camera is an infrared camera with a narrow infrared waveband of 940nm, and an infrared fill light is provided to fill light for the infrared camera.

[0029] S12: Two-dimensional code feature detection is performed on the sequence of image frames, and if the sequence of image frames contains a two-dimensional code, pose calculation is performed on the two-dimensional code feature to obtain the spatial position information of the two-dimensional code feature.

[0030] In this embodiment, there may be no two-dimensional code feature in the robot vision environment, there may be one two-dimensional code feature, or there may be multiple two-dimensional code features. Correspondingly, after obtaining the image information in the robot vision environment, some images in the sequence of image frames may have no two-dimensional code feature, some images may have one two-dimensional code feature, and some images may have multiple two-dimensional code features. In this embodiment, two-dimensional code feature detection can be performed on each image in the sequence of image frames to obtain all two-dimensional code features in the sequence of image frames. Each two-dimensional code feature has a corresponding two-dimensional code identifier, which is used to represent the identity, i.e. ID, of the two-dimensional code. In this embodiment, after obtaining all two-dimensional code features in the sequence of image frames, pose calculation can be performed on each two-dimensional code feature to obtain the spatial position information corresponding to each two-dimensional code feature. Specifically, the pose calculation process can be implemented by using a PNP (Perspective-n-Point) pose estimation algorithm.

[0031] In some embodiments of the present application, please refer to Figure 2 , Figure 2 A method implementation flowchart for obtaining the spatial position information of the two-dimensional code feature in the map construction method of the robot provided by the embodiments of the present application.

[0032] In detail:

[0033] S21: Two-dimensional code feature detection is performed on each image in the sequence of image frames, and it is judged whether the two-dimensional code exists in the image. If it exists, the pixel coordinates of the four corner points of the two-dimensional code feature are obtained.

[0034] S22: According to the pixel coordinates of the four corner points of the two-dimensional code feature, the relative distances of the four corner points of the two-dimensional code feature are determined, the two-dimensional code feature is subjected to pose estimation processing according to the relative distances, the position information of the four corner points of the two-dimensional code feature in the camera coordinate system is obtained, and the position information of the four corner points of the two-dimensional code feature in the camera coordinate system is determined as the spatial position information of the two-dimensional code feature.

[0035] In the embodiment, when it is detected that the two-dimensional code exists in the image, the pixel points representing the two-dimensional code extracted from the image can be obtained as the two-dimensional code feature, the two-dimensional code recognition is performed based on the two-dimensional code feature, and the two-dimensional code identifier corresponding to the two-dimensional code feature is obtained. According to the two-dimensional code identifier corresponding to the two-dimensional code feature and the pixel coordinates of the four corner points of the two-dimensional code feature, the identifier is assigned to the two-dimensional code feature, and each of the four corner points of the two-dimensional code feature is assigned a corresponding identifier. After obtaining the corresponding identifiers of the four corner points of the two-dimensional code feature, the pose of the camera relative to the two-dimensional code is obtained by using the PNP (Perspective-n-Point) pose estimation algorithm to calculate the pose of the two-dimensional code feature according to the relative distances of the four corner points of the two-dimensional code feature. It can be understood that the pose of the two-dimensional code is specifically represented by the position information of the four corner points in the camera coordinate system. It can be understood that the position information of each corner point in the camera coordinate system is mapped and associated with the identifier corresponding to the corner point. In the embodiment, the position information of the four corner points in the camera coordinate system is determined as the spatial position information of the two-dimensional code feature. It can be understood that when there are multiple two-dimensional code features in the image frame sequence, the spatial position information of the multiple two-dimensional code features can be obtained, and each two-dimensional code feature has corresponding identifier information.

[0036] S13: Feature points are extracted from the image frame sequence frame by frame, visual feature points of each image in the image frame sequence are obtained, and a corresponding feature descriptor is generated based on each visual feature point.

[0037] In this embodiment, visual feature points are found from the image frame sequence by extracting visual features from each image frame in the image frame sequence. For example, the Harris corner detection algorithm and / or the FAST feature point detection algorithm can be used to extract feature points from the image and obtain the visual feature points of each image in the image frame sequence. It will be appreciated that one or more visual feature points can be obtained from each image. The BRIEF (Binary Robust Independent Elementary Features) algorithm is a feature point descriptor algorithm. Specifically, a plurality of point pairs are randomly selected near a feature point, the gray values of the plurality of point pairs are combined into a binary string, and the binary string is determined as the feature descriptor of the feature point. In this embodiment, for each visual feature point obtained, the BRIEF algorithm can be used to select n point pairs around the visual feature point to generate a feature descriptor corresponding to each visual feature point.

[0038] S14: According to the visual feature points and the feature descriptors corresponding to the visual feature points, adjacent frame feature point matching is performed on the image frame sequence. If the visual feature points in two adjacent images match, the mileage information when the two adjacent images are captured is obtained, and the visual feature points in the two adjacent images are triangulated and depth estimated according to the mileage information to obtain the spatial position information of the visual feature points.

[0039] In this embodiment, the process of performing adjacent frame feature point matching on the image frame sequence can be as follows. According to the order of the images in the image frame sequence, the visual feature points obtained in a previous image and the feature descriptors corresponding to the visual feature points are compared with the visual feature points obtained in a subsequent image and the feature descriptors corresponding to the visual feature points. The similarity is compared, and it is determined whether the similarity reaches a preset similarity threshold. If the similarity between a visual feature point in the previous image and a visual feature point in the subsequent image and the similarity between the feature descriptor corresponding to the visual feature point in the previous image and the feature descriptor corresponding to the visual feature point in the subsequent image reach the preset similarity threshold, it is determined that the two visual feature points match. At this time, the displacement value between the two images can be calculated by using the kinematics of the robot to obtain the mileage information of the robot when the two images are captured, and the spatial position information of the visual feature points can be obtained by using the displacement value as an initial value to perform triangulation and depth estimation on the two matched visual feature points.

[0040] S15: According to the spatial position information of the visual feature points and the spatial position information of the two-dimensional code features, a map in the robot vision environment is constructed.

[0041] In this embodiment, after the spatial position information of the visual feature points and the spatial position information of the two-dimensional code features in the image frame sequence are obtained, the visual feature points and the two-dimensional code features are written into the map based on the spatial position information, and the visual feature points and the two-dimensional code features are distinguished by the markers, so that the map in the robot visual environment is constructed. It can be understood that, in the map, the spatial position information, the feature descriptor and the image feature position of the visual feature points are saved, and the spatial position information of the four corner points of the two-dimensional code features and the identifiers corresponding to the four corner points are saved

[0042] As can be seen above, the map construction method of the robot provided in the embodiments of the present application obtains the spatial position information of the two-dimensional code features and the spatial position information of the visual feature points in the image frame sequence by acquiring the image frame sequence by the camera, then performing two-dimensional code feature detection and visual feature point extraction on the image frame sequence, performing pose calculation on the two-dimensional code features and triangulation depth estimation on the visual feature points in combination with the mileage information. Then, the map in the robot visual environment is constructed according to the spatial position information of the visual feature points and the spatial position information of the two-dimensional code features. Based on the above method, the odometer information and the two-dimensional code information are fused to construct the map in the robot visual environment, so that the problem of easy loss of positioning of the mobile robot in scenes with many dynamic objects, low texture and poor lighting conditions can be solved.

[0043] In some embodiments of the present application, one visual feature point may appear in multiple consecutive image frames in the image frame sequence. In this embodiment, the spatial position information of all the visual feature points, the spatial position information of all the two-dimensional code features and the pose information when the camera acquires the image information are optimized by using the bundle adjustment algorithm on the multiple image frames in the image frame sequence, so as to obtain the optimized spatial position information of each visual feature point, the optimized spatial position information of each two-dimensional code feature and the optimized pose information of the camera. Then, the optimized spatial position information of each visual feature point and two-dimensional code feature is used for map construction, so as to improve the map accuracy. The bundle adjustment algorithm (Bundled Adjustment, English abbreviation BA) refers to adjusting the pose of the camera and the position of the feature points at the same time, so that the light reflected by each feature point passes through the optical center of the camera. Specifically, in this embodiment, the pose of the camera and the position of the feature points can be adjusted at the same time by using the re-projection minimization method.

[0044] In some embodiments of the present application, weight values can also be set for all visual feature points and all two-dimensional code features obtained from the image frame sequence, and the weight value corresponding to the visual feature point is set to be less than the weight value corresponding to the two-dimensional code feature. The set weight value is added to the optimization process of the spatial position information of the visual feature point and the spatial position information of the two-dimensional code feature, and the trust degree of the two-dimensional code feature in the constructed map is increased by taking the weight value as a constraint.

[0045] In some embodiments of the present application, please refer to Figure 3 , Figure 3 An implementation flowchart of a robot visual positioning method provided by an embodiment of the present application is provided. Details are as follows:

[0046] S31: loading a map in a robot visual environment according to preset feature point type weight configuration information;

[0047] S32: acquiring a real-time image in a current robot visual environment through the camera;

[0048] S33: detecting a two-dimensional code feature in the real-time image, and if the two-dimensional code feature is detected in the real-time image, obtaining a target two-dimensional code identifier corresponding to the two-dimensional code feature;

[0049] S34: traversing the map in the robot visual environment according to the two-dimensional code identifier, and if a two-dimensional code identifier matching the target two-dimensional code identifier is recorded in the map in the robot visual environment, performing visual positioning on the robot according to the two-dimensional code feature to obtain real-time position information of the robot.

[0050] In this embodiment, the feature point types include two types of visual feature point type and two-dimensional code feature type. A user can set feature point type weight configuration information for robot visual positioning in the robot according to actual needs. The higher the weight value corresponding to the configuration of the feature point type, the higher the confidence corresponding to the feature point of the feature point type in the map. In this embodiment, the weight value configured by the two-dimensional code feature type is greater than the weight value configured by the visual feature point type. When the robot performs visual positioning, the robot can load the map in the robot visual environment according to the preset feature point type weight configuration information, so that the feature points of different types in the map loaded by the robot have different confidences. After loading the map, the robot collects real-time images in the current visual environment of the robot through the camera, and then performs two-dimensional code feature detection on the real-time images to determine whether the real-time images have two-dimensional code features. If the real-time images have two-dimensional code features, the real-time position information of the robot is obtained by using the two-dimensional code features for PNP repositioning. Specifically, the target two-dimensional code identifier corresponding to the detected two-dimensional code features can be obtained by identifying the two-dimensional code features, and then the map loaded by the robot is traversed according to the two-dimensional code identifier. If the map records a two-dimensional code identifier matching the target two-dimensional code identifier, the coordinates of the current camera of the robot are converted to the map coordinate system, and then the robot is positioned visually according to the two-dimensional code features to obtain the real-time position information of the robot.

[0051] In some embodiments of the present application, please refer to Figure 4 , Figure 4 Another implementation flowchart of the robot visual positioning method provided by the embodiments of the present application is provided. Details are as follows:

[0052] S41: If no two-dimensional code features are detected in the real-time images, feature point extraction processing is performed on the real-time images to obtain target visual feature points and generate corresponding target feature point descriptors based on the target visual feature points;

[0053] S42: According to the target visual feature points and the target feature point descriptors corresponding to the target visual feature points, the map in the robot visual environment is traversed. If the map in the robot visual environment records visual feature points and target feature point descriptors corresponding to the visual feature points that match the target visual feature points and the target feature point descriptors corresponding to the target visual feature points, the robot is positioned visually according to the target visual feature points and the target feature point descriptors corresponding to the target visual feature points to obtain the real-time position information of the robot.

[0054] In this embodiment, when no two-dimensional code feature is detected in the real-time image after two-dimensional code feature detection is performed on the real-time image, visual positioning of the robot can be performed by matching visual feature points in the real-time image with visual feature points in the map. Specifically, target visual feature points in the real-time image are obtained by performing feature point extraction processing on the real-time image, and based on the target visual feature points, a feature descriptor corresponding to each target visual feature point is generated by selecting n point pairs around the target visual feature point as the center of the target visual feature point using the BRIEF algorithm. Further, according to the target visual feature points and the target feature point descriptors corresponding to the target visual feature points, the map loaded by the robot is traversed to determine whether visual feature points and visual feature point descriptors corresponding to the target visual feature points and the target feature point descriptors corresponding to the target visual feature points are recorded in the map. If visual feature points and visual feature point descriptors corresponding to the target visual feature points and the target feature point descriptors corresponding to the target visual feature points are recorded in the map, it indicates that feature matching is successful, at which time the coordinates of the current camera of the robot are converted to the map coordinate system, and the real-time position information of the robot is obtained by performing PNP repositioning using the matched visual feature points.

[0055] It can be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0056] In some embodiments of the present application, please refer to Figure 5 , Figure 5 is a basic structure block diagram of a robot map construction device provided by the embodiments of the present application. In this embodiment, each unit included in the device is used to perform each step in the above method embodiments. For details, please refer to the related description in the above method embodiments. For the sake of illustration, only the part related to this embodiment is shown. For example, Figure 5As shown in the figure, the map construction device of the robot comprises an acquisition module 51, a calculation module 52, an extraction module 53, an estimation module 54 and a mapping module 55. Among them: the acquisition module 51 is used for acquiring image information in the visual environment of the robot through the camera, and the image information is represented as a sequence of image frames. The calculation module 52 is used for two-dimensional code feature detection on the sequence of image frames, and if there is a two-dimensional code in the sequence of image frames, pose calculation is performed on the two-dimensional code feature to obtain the spatial position information of the two-dimensional code feature. The extraction module 53 is used for extracting feature points from the sequence of image frames frame by frame, acquiring the visual feature points of each image in the sequence of image frames, and generating a corresponding feature descriptor based on each visual feature point. The estimation module 54 is used for matching adjacent frame feature points according to the visual feature points and the feature descriptors corresponding to the visual feature points, and if the visual feature points in two adjacent images match, the mileage information when the two adjacent images are collected is obtained, and the spatial position information of the visual feature points is obtained according to the mileage information. The mapping module 55 is used for constructing a map in the visual environment of the robot according to the spatial position information of the visual feature points and the spatial position information of the two-dimensional code feature.

[0057] It should be understood that the above-mentioned map construction device of the robot corresponds to the above-mentioned map construction method of the robot, which will not be described here.

[0058] In some embodiments of the present application, please refer to Figure 6 , Figure 6 A basic structure block diagram of a visual positioning device of a robot is provided in some embodiments of the present application. In this embodiment, the units included in the device are used to perform the steps in the above-mentioned method embodiments. For details, please refer to the related description in the above-mentioned method embodiments. For the sake of illustration, only the parts related to this embodiment are shown. As Figure 6 shown, the visual positioning device of the robot comprises a loading module 61, an acquisition module 62, a detection module 63 and a positioning module 64. Among them: the loading module 61 is used for loading the map in the visual environment of the robot according to the pre-configured feature point type weight configuration information. The acquisition module 62 is used for acquiring real-time images in the current visual environment of the robot through the camera. The detection module 63 is used for two-dimensional code feature detection on the real-time images, and if a two-dimensional code feature is detected in the real-time images, a target two-dimensional code identifier corresponding to the two-dimensional code feature is obtained. The positioning module 64 is used for traversing the map in the visual environment of the robot according to the two-dimensional code identifier, and if a two-dimensional code identifier matching the target two-dimensional code identifier is recorded in the map in the visual environment of the robot, the robot is visually positioned according to the two-dimensional code feature to obtain real-time position information of the robot.

[0059] It should be understood that the visual positioning device of the robot described above corresponds to the visual positioning method of the robot described above, and will not be described here.

[0060] In some embodiments of the present application, please refer to Figure 7 , Figure 7 A basic structure block diagram of an electronic device is provided for embodiments of the present application. As shown in Figure 7 The electronic device 7 of the embodiment includes a processor 71, a memory 72, and a computer program 73 stored in the memory 72 and executable on the processor 71, such as a program of the robot map construction method or a program of the robot visual positioning method. The processor 71 implements the steps of the robot map construction method or the robot visual positioning method described above when executing the computer program 73. Alternatively, the processor 71 implements the functions of each module in the corresponding embodiment of the robot map construction device or the robot visual positioning device when executing the computer program 73. For details, please refer to the related description in the embodiments, which will not be described here.

[0061] For example, the computer program 73 can be divided into one or more modules (units) stored in the memory 72 and executed by the processor 71 to complete the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 73 in the electronic device 7. For example, the computer program 73 can be divided into an acquisition module, a calculation module, an extraction module, an estimation module, and a mapping module, or divided into a loading module, an acquisition module, a detection module, and a positioning module, and the functions of each module are as described above.

[0062] The electronic device can include, but is not limited to, the processor 71 and the memory 72. Those skilled in the art can understand that Figure 7 The electronic device 7 is only an example and does not constitute a limitation on the electronic device 7, and can include more or fewer components than shown, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, etc.

[0063] The processor 71 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0064] The memory 72 can be an internal storage unit of the electronic device 7, such as a hard disk or a memory of the electronic device 7. The memory 72 can also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 72 can include both the internal storage unit and the external storage device of the electronic device 7. The memory 72 is used to store the computer program and other programs and data required by the electronic device. The memory 72 can also be used to temporarily store data that has been output or will be output.

[0065] It should be noted that the information interaction, execution process, etc. between the above apparatuses / units, since based on the same concept as the method embodiments, the specific functions and the brought technical effects can be referred to the method embodiments part, and will not be repeated here.

[0066] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned method embodiments. In the embodiments, the computer readable storage medium can be non-volatile or volatile.

[0067] The embodiments of the present application provide a computer program product, which, when running on a mobile terminal, enables the mobile terminal to implement the steps in the above-mentioned method embodiments.

[0068] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software function unit. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0069] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code. It should be noted that the contents included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0070] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0071] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A map-building method for a robot, characterized in that, include: Image information of the robot's visual environment is acquired through a camera, and the image information is represented as a sequence of image frames. The image frame sequence is subjected to QR code feature detection. If a QR code exists in the image frame sequence, the pose of the QR code feature is calculated to obtain the spatial location information of the QR code feature. Feature points are extracted frame by frame from the image frame sequence to obtain the visual feature points of each frame in the image frame sequence and a corresponding feature descriptor is generated based on each visual feature point. Based on the visual feature points and the feature descriptors corresponding to the visual feature points, the image frame sequence is matched with adjacent frame feature points. If the visual feature points in two adjacent frames match, the mileage information at the time of acquisition of the two adjacent frames is obtained. The displacement value between the two frames is calculated using the mileage information. The displacement value is used as the initial value to perform triangulation depth estimation on the visual feature points in the two adjacent frames to obtain the spatial position information of the visual feature points. Based on spatial location information, the visual feature points and the QR code features are written into the map, and the visual feature points and the QR code features are distinguished by markings to construct a map of the robot's visual environment; Before constructing the map of the robot's visual environment by writing the visual feature points and the QR code features into the map based on spatial location information and distinguishing the visual feature points and the QR code features by marking them, the method further includes: setting weight values ​​for the visual feature points and the QR code features, wherein the weight value corresponding to the visual feature points is less than the weight value corresponding to the QR code features; and using a bundle adjustment algorithm to optimize the spatial location information of the visual feature points, the spatial location information of the QR code features, and the pose information of the camera by combining multiple frames in the image frame sequence.

2. The robot map construction method according to claim 1, characterized in that, The step of performing QR code feature detection on the image frame sequence, and if a QR code exists in the image frame sequence, calculating the pose of the QR code features to obtain the spatial location information of the QR code features, includes: Perform QR code feature detection on each frame in the image frame sequence to determine whether a QR code exists in the image. If it exists, obtain the pixel coordinates of the four corner points of the QR code feature. Based on the pixel coordinates of the four corner points of the QR code feature, the relative distance between the four corner points of the QR code feature is determined. Based on the relative distance, pose estimation processing is performed on the QR code feature to obtain the position information of the four corner points of the QR code feature in the camera coordinate system. The position information of the four corner points of the QR code feature in the camera coordinate system is determined as the spatial position information of the QR code feature.

3. A visual localization method for a robot, characterized in that, The robot's visual localization method uses a map constructed by the robot's map building method as described in any one of claims 1-2 for visual localization, and the robot's visual localization method includes: The map in the robot's visual environment is loaded according to the preset feature point type weight configuration information; The camera captures real-time images of the robot's current visual environment. The real-time image is subjected to QR code feature detection. If QR code features are detected in the real-time image, the target QR code identifier corresponding to the QR code features is obtained. Based on the QR code identifier, the map in the robot's visual environment is traversed. If a QR code identifier matching the target QR code identifier is recorded in the map of the robot's visual environment, the robot is visually located based on the QR code features to obtain the robot's real-time location information.

4. The visual positioning method for a robot according to claim 3, characterized in that, After the step of performing QR code feature detection on the real-time image, the method further includes: If no QR code features are detected in the real-time image, feature point extraction processing is performed on the real-time image to obtain target visual feature points and generate corresponding target feature point descriptors based on the target visual feature points; Based on the target visual feature point and the target feature point descriptor corresponding to the target visual feature point, the map in the robot's visual environment is traversed. If the map in the robot's visual environment records a visual feature point and a target feature point descriptor corresponding to the target visual feature point and the target feature point descriptor corresponding to the target visual feature point, then the robot is visually located based on the target visual feature point and the target feature point descriptor corresponding to the target visual feature point to obtain the robot's real-time position information.

5. A map-building device for a robot, characterized in that, include: The acquisition module is used to acquire image information in the robot's visual environment through a camera, wherein the image information is represented as an image frame sequence; The calculation module is used to perform QR code feature detection on the image frame sequence. If a QR code exists in the image frame sequence, the module performs pose calculation on the QR code feature to obtain the spatial location information of the QR code feature. The extraction module is used to extract feature points frame by frame from the image frame sequence, obtain the visual feature points of each frame in the image frame sequence, and generate a corresponding feature descriptor based on each visual feature point. The estimation module is used to perform adjacent frame feature point matching on the image frame sequence based on the visual feature points and the feature descriptors corresponding to the visual feature points. If the visual feature points in two adjacent frames match, the mileage information at the time of acquisition of the two adjacent frames is obtained. The displacement value between the two frames is calculated using the mileage information. The displacement value is used as the initial value to perform triangulation depth estimation on the visual feature points in the two adjacent frames to obtain the spatial position information of the visual feature points. The mapping module is used to write the visual feature points and the QR code features into a map based on spatial location information, and to distinguish the visual feature points and the QR code features by marking them, thereby constructing a map in the robot's visual environment. The map building device is further configured to, before writing the visual feature points and the QR code features into the map based on spatial location information, and distinguishing the visual feature points and the QR code features by marking them to construct the map in the robot's visual environment, set weight values ​​for the visual feature points and the QR code features, wherein the weight value corresponding to the visual feature points is less than the weight value corresponding to the QR code features; and combine multiple frames in the image frame sequence to optimize the spatial location information of the visual feature points, the spatial location information of the QR code features, and the pose information of the camera using a bundle adjustment algorithm.

6. A visual positioning device for a robot, characterized in that, The robot's visual positioning device performs visual positioning using a map constructed by the robot's map building method as described in any one of claims 1-2, including: The loading module is used to load the map in the robot's visual environment according to the preset feature point type weight configuration information; The acquisition module is used to acquire real-time images of the robot's current visual environment through the camera; The detection module is used to perform QR code feature detection on the real-time image. If QR code features are detected in the real-time image, the target QR code identifier corresponding to the QR code features is obtained. The positioning module is used to traverse the map in the robot's visual environment based on the QR code identifier. If the map in the robot's visual environment records a QR code identifier that matches the target QR code identifier, then the robot is visually positioned based on the QR code features to obtain the robot's real-time position information.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-2 or 3-4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-2 or 3-4.

Images