Multi-robot visual positioning method and device, storage medium and computer device

Abstract

The application discloses a visual robot pose estimation method and device, a storage medium and a computer device, and relates to the technical field of robot positioning. The method comprises the following steps: a visual robot extracts visual point features of a current visual image, and extracts shared point features and shared surface features in a shared map; the visual point features of the current visual image are matched with the shared point features and the shared surface features in the shared map respectively, so that the shared point features and the shared surface features which have a matching relationship with the visual point features are obtained; according to the shared point features and the shared surface features which have the matching relationship, a first residual Jacobian matrix of the shared point features to the visual point features and a second residual Jacobian matrix of the shared surface features to the visual point features are constructed respectively; and according to the first residual Jacobian matrix and the second residual Jacobian matrix, a target camera pose of the current visual image is solved by using a G2O library.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot positioning, in particular to a multi-robot visual positioning method and device, a storage medium and a computer device. BACKGROUND

[0002] In the prior art, according to the different contents of the shared map, the visual positioning algorithm of the mobile robot based on the shared map is divided into two categories:

[0003] The first category is that the shared map is a three-dimensional point cloud containing visual features, and the feature matching based on the shared map is divided into two kinds: 1) 3D-2D feature matching. The three-dimensional point features and surface features in the shared map are projected onto the pixel plane and matched with the camera observation features; 2) 3D-3D feature matching. The three-dimensional feature information perceived by the visual sensor is matched with the three-dimensional features of the shared map. However, it is difficult to establish a real-time association between the features extracted by the visual positioning algorithm of the mobile robot using the shared map and the laser radar features in the map, and false matching is prone to occur.

[0004] The second category is that the shared map is a laser radar map without related visual features, and the color point cloud projection and the synthesized depth image are used to match the real-time image for camera positioning. This positioning scheme converts the two-dimensional to three-dimensional matching problem into a two-dimensional to two-dimensional matching problem, but the false matching is more common in the visual positioning algorithm of the mobile robot using the shared map, which affects the positioning accuracy of the robot. SUMMARY

[0005] Therefore, the present application provides a multi-robot visual positioning method and device, a storage medium and a computer device, which mainly aims to solve the problem that the association between the visual features and the laser radar features is difficult to establish and false matching is prone to occur, and to reduce the problem of reduced robot positioning accuracy caused by reduced matching dimension.

[0006] According to one aspect of the present application, a multi-robot visual positioning method is provided, which comprises:

[0007] extracting visual point features of a current visual image by a visual robot, and shared point features and shared surface features in a shared map;

[0008] matching the visual point features of the current visual image with the shared point features and the shared surface features in the shared map respectively to obtain shared point features and shared surface features having a matching relationship with the visual point features;

[0009] constructing a first residual Jacobian matrix of the shared point features to the visual point features and a second residual Jacobian matrix of the shared surface features to the visual point features according to the shared point features and the shared surface features having the matching relationship.

[0010] solving the target camera pose of the current visual image according to the first residual Jacobian matrix and the second residual Jacobian matrix by using a G2O library.

[0011] According to another aspect of the present application, a multi-robot visual positioning device is provided, which comprises:

[0012] an extraction module configured to extract visual point features of a current visual image of a visual robot, and shared point features and shared face features in a shared map;

[0013] a matching module configured to match the visual point features of the current visual image with the shared point features and the shared face features in the shared map respectively, to obtain shared point features and shared face features having a matching relationship with the visual point features;

[0014] a construction module configured to construct a first residual Jacobian matrix of the shared point features to the visual point features and a second residual Jacobian matrix of the shared face features to the visual point features according to the shared point features and the shared face features having the matching relationship;

[0015] an estimation module configured to solve a target camera pose of the current visual image according to the first residual Jacobian matrix and the second residual Jacobian matrix by using a G2O library.

[0016] According to still another aspect of the present application, a storage medium having a computer program stored thereon is provided, the program being executed by a processor to implement the multi-robot visual positioning method.

[0017] According to yet another aspect of the present application, a computer device is provided, which comprises a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, the processor implementing the multi-robot visual positioning method when executing the program.

[0018] By the above technical solution, the multi-robot visual positioning method and device, storage medium and computer device provided by the present application are compared with the prior technical solution of difficulty in establishing a feature matching association relationship for a visual feature and the technical solution of dimension reduction matching, and the visual robot extracts visual point features of a current visual image and shared point features and shared face features in a shared map; the visual point features of the current visual image and the shared point features and shared face features in the shared map are respectively matched to obtain shared point features and shared face features having a matching relationship with the visual point features; a first residual Jacobian matrix of the shared point features to the visual point features and a second residual Jacobian matrix of the shared face features to the visual point features are constructed according to the shared point features and shared face features having the matching relationship; and a target camera pose of the current visual image is solved by using a G2O library according to the first residual Jacobian matrix and the second residual Jacobian matrix. It can be seen that the visual rough positioning realized by the shared point feature to visual point feature matching and the precise positioning of the robot realized by the shared face feature to visual point feature matching are beneficial to the visual robot to directly match the visual features, reduce the difficulty of the association of the visual features and the radar features, increase the endurance of the robot, reduce the cost and self-weight of the robot, and increase the working efficiency of the robot.

[0019] The above description is only a summary of the technical solutions of the present application. In order to enable one skilled in the art to better understand the technical means of the present application, the content of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present application to be more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0020] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 A flowchart of a multi-robot visual positioning method provided by an embodiment of the present application is shown;

[0022] Figure 2 A flowchart of another multi-robot visual positioning method provided by an embodiment of the present application is shown;

[0023] Figure 3 A flowchart of point feature extraction and matching provided by an embodiment of the present application is shown;

[0024] Figure 4 A structural diagram of a multi-robot visual positioning device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] Hereinafter, the present application will be described in detail with reference to the accompanying drawings and embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0026] In view of the fact that the correlation between the existing visual features and the lidar features is difficult to establish, and the robot positioning accuracy is reduced due to the reduction of the matching dimension, the present embodiment provides a multi-robot visual positioning method, and a multi-robot visual positioning algorithm based on a shared map in a ship deck environment is designed. The present embodiment requires a small number of mobile robots to carry two sensors, i.e., a laser radar and a camera. The visual robot (working robot) realizes its own positioning work based on the working environment map (shared map) formed by the mobile robot, thereby effectively avoiding the above technical problems existing in the prior art, reducing the difficulty of correlating visual features and radar features, increasing the endurance of the robot, reducing the cost and self-weight of the robot, and increasing the working efficiency of the robot. As shown in FIG. Figure 1 The method comprises the following steps.

[0027] 101. The visual robot extracts the visual point features of the current visual image, and the shared point features and shared face features in the shared map.

[0028] In the present embodiment, the shared map is established by a mobile robot carrying a camera and a laser radar sensor. The shared map is a shared map loaded offline by the mobile robot, and the visual robot is a mobile robot carrying a visual sensor. The shared map contains visual features and accurate depth information, which is beneficial to direct matching of visual features based on the visual robot, thereby reducing the difficulty of correlating visual features and radar features, increasing the endurance and working efficiency of the robot in the ship deck environment, and reducing the cost and self-weight of the robot.

[0029] Among them, the map is loaded offline before the mobile robot works, that is, the offline map is established by the mobile robot. The offline map provides visual features and accurate depth information for a mobile robot carrying only a visual sensor, thereby realizing the reuse of the shared map.

[0030] 102. The visual point features of the current visual image are matched with the shared point features and shared face features in the shared map respectively, to obtain the shared point features and shared face features having a matching relationship with the visual point features.

[0031] In the embodiment, the shared map-based mobile robot visual positioning algorithm is to obtain the pose information of the visual robot by matching the visual point features of the current visual image with the shared point features and the shared surface features in the shared map respectively, so as to effectively cope with the influence of light, shadow and day and night alternation on the visual robot in the ship deck environment. At the same time, in order to reduce the time cost of the algorithm, the shared point feature to visual point feature matching and the shared surface feature to visual point feature matching are designed, that is, the feature matching positioning strategy from coarse to fine is designed, so as to avoid the problem of low positioning accuracy of single visual feature.

[0032] 103. Constructing a first residual Jacobian matrix of the shared point feature to visual point feature and a second residual Jacobian matrix of the shared surface feature to visual point feature according to the shared point feature and the shared surface feature having the matching relationship.

[0033] 104. Solving the target camera pose of the current visual image by using the G2O library according to the first residual Jacobian matrix and the second residual Jacobian matrix.

[0034] In the embodiment, the Jacobian matrix of the point feature re-projection error is obtained by the residual construction of the shared point feature to visual point feature, and the Jacobian matrix for further optimizing the pose is obtained by the residual construction of the shared surface feature to visual point feature. Specifically, based on the least square problem of the shared point feature to visual point feature and the matching error of the shared surface feature to visual point feature, an optimization loss function is constructed, and the corresponding Jacobian matrix is derived by the chain rule, so as to further solve the optimal camera pose by using the loss function graph in the G2O library.

[0035] For the embodiment, the visual robot can extract the visual point features of the current visual image and the shared point features and the shared surface features in the shared map according to the above-mentioned scheme; the visual point features of the current visual image are matched with the shared point features and the shared surface features in the shared map respectively to obtain the shared point features and the shared surface features having the matching relationship with the visual point features; the first residual Jacobian matrix of the shared point feature to visual point feature and the second residual Jacobian matrix of the shared surface feature to visual point feature are constructed according to the shared point features and the shared surface features having the matching relationship; and the target camera pose of the current visual image is solved by using the G2O library according to the first residual Jacobian matrix and the second residual Jacobian matrix. It can be seen that the visual coarse positioning realized by the shared point feature to visual point feature matching and the precise positioning of the robot realized by the shared surface feature to visual point feature matching are beneficial to the visual robot to directly match the visual features, reduce the difficulty of associating the visual features and the radar features, increase the endurance of the robot, reduce the cost and self-weight of the robot, and increase the working efficiency of the robot.

[0036] Further, as a refinement and extension of the above embodiment, in order to complete the description of the specific implementation process of the embodiment, another multi-robot visual positioning method is provided, as shown in the figure, the method comprises: Figure 2

[0037] 201, the visual robot extracts the visual point features of the current visual image, and shares the shared point features and shared face features in the shared map.

[0038] 202, match the visual point features of the current visual image with the shared point features in the shared map, and obtain the shared point features that have a matching relationship with the visual point features.

[0039] In order to illustrate the specific implementation of step 202, the visual point features include the extracted visual Fast corner points and their visual Brief descriptors, and the shared point features in the shared map include the extracted shared Fast corner points and their shared Brief descriptors. As a preferred embodiment, step 202 can specifically include: according to the visual Brief descriptor and the shared Brief descriptor, using the bag-of-words model for feature matching, and through angle consistency filtering processing, obtaining the shared Brief descriptor that has a matching relationship with the visual Brief descriptor; based on the correspondence between the Fast corner points and the Brief descriptors, obtaining the shared Fast corner point that has a matching relationship with the visual Fast corner point.

[0040] In implementation, based on the ORB point features, the bag-of-words model is used for visual rough positioning of the shared map-based visual robot, that is, according to the ORB point features of the current visual image, the Fast corner points and Brief descriptors of the ORB point features are extracted, the visual point features are the extracted Fast corner points, and based on the extracted visual Fast corner points of the current visual image, the corresponding visual Brief descriptors are matched with the shared Brief descriptors of the shared Fast corner points in the shared map using the bag-of-words model. It should be noted that the ORB point features are used as the feature information for motion estimation, and the selection criteria for the Fast corner points is: if a pixel point has a large difference in brightness with its surrounding pixel points, it is considered as a Fast corner point, and the gray centroid method is used to add quantitative description of rotation and angle to the corner point, and the image pyramid is introduced to avoid false extraction due to scale problems.

[0041] Further, the point feature matching is the matching of the descriptors carried by the Fast corner points, and the formation of the descriptors of the point features is based on the surrounding pixels, that is, a vector composed of 0 and 1 is obtained by comparing the brightness of the target pixel, as shown in the figure. Figure 3 The point feature extraction and matching process is shown in the figure.​

[0042] 203、matching the visual point features of the current visual image with the shared face features in the shared map to obtain shared face features having a matching relationship with the visual point features.

[0043] To illustrate the specific implementation of step 203, as a preferred embodiment, step 203 can specifically include: eliminating outliers in the shared face features in the shared map; and matching the visual point features of the current visual image with the shared face features from which the outliers are eliminated based on the normal vectors of the shared face features to obtain shared face features having a matching relationship with the visual point features.

[0044] In implementation, the shared face features are extracted according to the 3D point cloud data in the shared map. Since the 3D point cloud information has relatively accurate depth information, the extraction of the shared face features can be realized by using the depth information, and the situation of severe light change can be better coped with. The specific shared face feature extraction method includes a region generation algorithm, a Hough transform algorithm and a random sample consensus algorithm.

[0045] Further, the embodiment is applied to the environment in which a ship deck visual robot works. The single feature association based on visual positioning cannot meet the tracking demand of the visual robot features, and the visual positioning of the visual robot based on the shared map needs to be performed under a large visual angle. Therefore, the global pose estimation of the visual robot is further realized by combining the point-face feature association mode.

[0046] The feature association strategy of the visual robot in the ship deck environment is specifically that the threshold is set to distinguish the inner points and the outer points in the point cloud data. The inner point is a point whose distance from the plane model is less than the set threshold. Similarly, the outliers as the outer points can be found, and the outliers are eliminated from the point cloud data. A group of model parameters are found by using the following formula (1) so that the formula (1) determined by the group of parameters can represent the number of points on the plane exceeding the set threshold.

[0047] (1)

[0048] wherein, n represents the unit normal vector of the feature , is the distance of the feature to (0, 0, 0).

[0049] 204、constructing a first residual Jacobian matrix of the shared point features to the visual point features according to the shared point features having the matching relationship.

[0050] To illustrate the specific implementation of step 204, as a preferred embodiment, step 204 may specifically include: determining the second pixel coordinates of the target point in the current visual image at time k+1 based on the first pixel coordinates of the target point in the current visual image at time k; constructing a least squares problem from the shared point features to the visual point features based on the first pixel coordinates and the second pixel coordinates; and obtaining the first residual Jacobian matrix from the shared point features to the visual point features based on the least squares problem.

[0051] In implementation, to better address the requirement of accurate vision-based localization for visual robots on ship decks, a strategy of optimizing the loss function using point-to-point features and point-to-surface feature matching to solve for robot pose is adopted. The specific construction of the residual from shared point features to visual point features is as follows:

[0052] Take a point in space based on the camera imaging model. Based on the camera's intrinsic parameters, the following information is obtained: Time and Space Pixel coordinates on the camera's imaging plane When the movement reaches Time, space, point After matching the brief descriptors of the ORB point features, the pixel coordinates on the current camera plane are determined. ,in, The pixel coordinates at time can be determined based on The pixel coordinates at each time point and the pose transformation between frames are represented as shown in equation (2):

[0053]

[0054] in, For camera internal parameters, For camera Time's up Inter-frame pose transformation at time step. Pixel coordinates are obtained from the transformation matrix. Time and Space Point The pixel coordinates of the projection are used to construct a least-squares problem, as shown in equation (3):

[0055]

[0056] According to the chain rule, spatial features are transformed to the camera coordinate system through calibration extrinsic parameters. reprojection error Relative to camera pose First Jacobian Matrix for:

[0057]

[0058] where the first term is the derivative of the error with respect to the projection point According to the observed three-dimensional coordinates in the camera coordinate system, the pixel coordinates are converted according to the camera intrinsic parameters, as shown in equations (5) and (6):

[0059]

[0060] where, According to the camera imaging model, the following can be obtained:

[0061]

[0062] The second term in equation (4) is the derivative of the three-dimensional point feature in the camera coordinate system with respect to the Lie algebra:

[0063]

[0064] where, represents a one-time external parameter transformation of the spatial point P corresponding to the Lie algebra, is a small disturbance left multiplied by the derivative of the pose. Multiplying the two terms gives the first Jacobian matrix of the point feature re-projection error:

[0065]

[0066] 205. According to the shared face feature with a matching relationship, a second residual Jacobian matrix of the shared face feature to the visual point feature is constructed.

[0067] To illustrate the specific implementation of step 205, as a preferred embodiment, step 205 can specifically include: based on the normal vector of the shared face feature, constructing a matching error of the shared face feature to the visual point feature; according to the matching error, obtaining the second residual Jacobian matrix of the shared face feature to the visual point feature through partial derivative processing.

[0068] The residual construction of the shared face feature to the visual point feature is specifically: according to the point-to-plane distance, a loss function is established, as shown in equation (11), represents the matching error of the 3D feature point extracted from the current visual image and the 3D face feature in the shared map, and the Jacobian matrix is obtained through chain derivation, as shown in equation (12).

[0069]

[0070] where, represents the error matched in the camera coordinate system, is the representation of the plane feature extracted from the shared map in the camera coordinate system. The point feature extracted from the current visual image The error of matching with the plane is established by the normal vector of the plane, so represents the partial derivative of the error function to the normal vector of the plane feature in the camera coordinate system of the shared map, as shown in equation (13):

[0071]

[0072] wherein, represents the plane feature in the world coordinate system in the shared map. The first dimension in equation (20) is obtained as shown in equation (18):

[0073]

[0074] Therefore, according to equation (13) and equation (18), the plane matching error is obtained, that is, the Jacobian matrix for optimizing the pose:

[0075]

[0076] 206. According to the first residual Jacobian matrix and the second residual Jacobian matrix, the target camera pose of the current visual image is solved by using the G2O library.

[0077] In order to illustrate the specific embodiment of step 206, as a preferred embodiment, step 206 can specifically include: constructing a target loss function according to the least square problem and the matching error; constructing a total Jacobian matrix for the target loss function according to the first residual Jacobian matrix and the second residual Jacobian matrix; solving the target camera pose of the current visual image by using the G2O library according to the total Jacobian matrix; wherein the target loss function includes the matching scores of the co-visible keyframes of the candidate keyframes in the shared map and the current visual image.

[0078] In the implementation, after extracting the point and surface features, a target loss function is established according to the point and surface features, and the optimal matching is selected through the score of the target loss function, that is, the optimal extrinsic parameter value of the visual robot is obtained by optimizing the matching errors of multiple features. The target loss function includes the matching scores of the co-visible keyframes of the candidate keyframes and the current visual image, which is more accurate than estimating the camera pose only by matching the candidate keyframes and the current visual image, avoids the contingency of single-frame matching, and prevents the visual robot from falling into local optimum based on the shared map positioning. A fixed-size window is set before the positioning of the visual robot, which reduces the calculation amount of the positioning algorithm and saves the time cost of the positioning algorithm.

[0079] ​Specifically, a local window is selected for image matching, the local window contains a pre-set number of co-visible keyframes selected from fixed candidate keyframes, and a plurality of camera state quantities of a shared robot in a shared map, a current position of the visual robot outside the local window, i.e., a camera state quantity of a current visual image, and then an optimal position of the current visual image of the visual robot is solved through feature matching of the current visual image and the candidate keyframes and features of the co-visible keyframes of the candidate keyframes in the local window. Wherein, the relative extrinsic parameter transformation between the candidate keyframes and the co-visible keyframes is pre-stored in the shared map, so the relative extrinsic parameter transformation between the current visual image and the candidate keyframes can be represented.

[0080] The camera pose is solved based on point feature and surface feature matching, and the overall optimization target loss function of point-point feature and point-surface feature matching error is shown in formula (20):

[0081]

[0082] According to the target loss function, the matching scores of all candidate keyframes and the current visual image in each local window are calculated, so as to determine the local window with the highest matching score, and then find a keyframe with the highest matching score from the matched candidate keyframes in the local window, calculate the extrinsic parameter transformation between the current visual image and the keyframe, and thus the position of the current visual image relative to the world coordinate system is solved.

[0083] The target loss function is established as shown in formula (20), including the reprojection error of the feature points obtained by matching the current visual image with the feature points in the shared map and the visual feature point error obtained by matching the surface features in the shared map. The total Jacobian matrix of the target loss function is obtained from the point feature to point feature residual Jacobian matrix of formula (10) , the point feature and surface feature matching residual Jacobian matrix of formula (19) .

[0084]

[0085] According to the obtained total Jacobian matrix, the optimal camera pose is solved by using the G2O library to optimize the loss function graph. According to the requirements of the actual application scene, the Gauss-Newton method can be used to solve the loss function.

[0086] By applying the technical solution of the embodiment, the ORB feature points are used to perform visual rough positioning of the visual robot based on the shared map by using the bag-of-words model, possible local positions of the visual robot in the map are found, then the pose estimation method of fusing point and surface features is used, the point and surface features are jointly matched and positioned, and the visual robot is accurately positioned based on the matching strategy of the local map window to be matched, so as to avoid the contingency of the single-frame matching positioning based on the visual features.

[0087] Further, as Figure 1 In a specific implementation of the method, the embodiment of the present application provides a multi-robot visual positioning device, as shown in the figure, which comprises an extraction module 41, a matching module 42, a construction module 43, and an estimation module 44. Figure 4 The extraction module 41 is configured to extract visual point features of a current visual image of a visual robot, and shared point features and shared face features in a shared map.

[0088] The extraction module 41 is configured to extract visual point features of a current visual image of a visual robot, and shared point features and shared face features in a shared map.

[0089] The matching module 42 is configured to match the visual point features of the current visual image with the shared point features and the shared face features in the shared map respectively, to obtain shared point features and shared face features that have a matching relationship with the visual point features.

[0090] The construction module 43 is configured to construct a first residual Jacobian matrix of the shared point features to the visual point features and a second residual Jacobian matrix of the shared face features to the visual point features according to the shared point features and the shared face features that have a matching relationship.

[0091] The estimation module 44 is configured to solve a target camera pose of the current visual image by using a G2O library according to the first residual Jacobian matrix and the second residual Jacobian matrix.

[0092] In a specific application scenario, the shared map is a shared map loaded offline by a mobile robot, the mobile robot is a mobile robot carrying a camera and a laser radar sensor, and the visual robot is a mobile robot carrying a visual sensor.

[0093] In a specific application scenario, the visual point features include extracted visual Fast corner points and visual Brief descriptors thereof, the shared point features in the shared map include extracted shared Fast corner points and shared Brief descriptors thereof, and the matching module 42 includes a first matching unit 421 and a second matching unit 422.

[0094] The first matching unit 421 is configured to perform feature matching by using a bag-of-words model according to the visual Brief descriptors and the shared Brief descriptors, and to obtain shared Brief descriptors that have a matching relationship with the visual Brief descriptors by angle consistency filtering processing; and to obtain shared Fast corner points that have a matching relationship with the visual Fast corner points based on a corresponding relationship between the Fast corner points and the Brief descriptors.

[0095] The second matching unit 422 is configured to remove outliers in the shared face features in the shared map, and match the visual point features in the current visual image with the shared face features after removing the outliers based on the normal vectors of the shared face features, to obtain shared face features having a matching relationship with the visual point features.

[0096] In a specific application scenario, the construction module 43 includes a first construction unit 431 and a second construction unit 432.

[0097] The first construction unit 431 is configured to determine second pixel coordinates of a target point in a k+1 time current visual image based on first pixel coordinates of the target point in a k time current visual image, and construct a least square problem of the shared point features to the visual point features according to the first pixel coordinates and the second pixel coordinates, to obtain a first residual Jacobian matrix of the shared point features to the visual point features according to the least square problem.

[0098] The second construction unit 432 is configured to construct a matching error of the shared face features to the visual point features based on the normal vectors of the shared face features, and obtain a second residual Jacobian matrix of the shared face features to the visual point features through partial derivative processing according to the matching error.

[0099] In a specific application scenario, the estimation module 44 is specifically configured to construct a target loss function according to the least square problem and the matching error, construct a total Jacobian matrix for the target loss function according to the first residual Jacobian matrix and the second residual Jacobian matrix, and solve a target camera pose of the current visual image by using a G2O library according to the total Jacobian matrix, wherein the target loss function includes matching scores of co-view key frames of candidate key frames in the shared map and the current visual image.

[0100] It should be noted that other corresponding descriptions of the functions of the multi-robot visual positioning device provided in the embodiments of the present application can refer to the corresponding descriptions in Figure 1 and Figure 2 , which will not be described here in detail.

[0101] Based on the above method as shown in Figure 1 and Figure 2 , correspondingly, the embodiments of the present application also provide a storage medium having a computer program stored thereon, and the program is executed by a processor to implement the multi-robot visual positioning method as shown in Figure 1 and Figure 2 .

[0102] Based on the understanding, the technical scheme of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in various implementation scenarios of the present application.

[0103] Based on the above method as shown in Figure 1 , Figure 2 and the virtual device embodiment as shown in Figure 4 , in order to achieve the above purpose, the embodiments of the present application also provide a computer device, which can be a personal computer, a server, a network device, etc., and the entity device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to realize the multi-robot visual positioning method as shown in Figure 1 and Figure 2 .

[0104] Optionally, the computer device can also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface can include a display screen (Display), an input unit such as a keyboard (Keyboard), etc. The optional user interface can also include a USB interface, a card reader interface, etc. The network interface can optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0105] Those skilled in the art can understand that the computer device structure provided by the embodiments does not constitute a limitation on the entity device, which can include more or fewer components, or combine certain components, or different component arrangements.

[0106] The storage medium can also include an operating system, a network communication module. The operating system is a program that manages computer device hardware and software resources, supports information processing programs and the running of other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the entity device.

[0107] Those skilled in the art can clearly understand the application can be realized by means of software plus necessary general hardware platform, also can be realized by hardware through the description of the above embodiments. By applying the technical solutions of the application, compared with the prior art, the technical solutions of the present application can be used for establishing the association relationship of the feature matching containing visual features, and the technical solutions of the present application can avoid the difficulty of matching and the error matching. In the present embodiment, the ORB feature points are used to find the possible local positions of the visual robot in the map by using the bag-of-words model for visual robot visual rough positioning based on the shared map. Then, the pose estimation method of the fusion point and surface feature is used to jointly match and position the point and surface features, and the visual robot is accurately positioned based on the matching strategy of the local map window to be matched, so as to avoid the contingency of the single frame matching positioning based on the visual feature.

[0108] Those skilled in the art can understand that the modules or processes in the drawings are not necessarily required for the implementation of the present application. Those skilled in the art can understand that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the description of the implementation scenario, or can be changed and located in one or more devices different from the implementation scenario. The modules of the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0109] The above application serial number is only for description, not representing the advantages and disadvantages of the implementation scenario. The above disclosure is only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the present application.

Claims

1. A multi-robot visual positioning method, characterized by, The method comprises the steps of: extracting visual point features of a current visual image by a visual robot, and sharing point features and shared face features in a shared map; matching the visual point features of the current visual image with the shared point features and the shared face features in the shared map respectively to obtain shared point features and shared face features having a matching relationship with the visual point features; wherein the visual point features comprise extracted visual Fast corner points and visual Brief descriptors thereof, the shared point features in the shared map comprise extracted shared Fast corner points and shared Brief descriptors thereof, feature matching is performed by using a bag-of-words model according to the visual Brief descriptors and the shared Brief descriptors, and shared Brief descriptors having a matching relationship with the visual Brief descriptors are obtained through angle consistency filtering processing; based on the correspondence between the Fast corner points and the Brief descriptors, shared Fast corner points having a matching relationship with the visual Fast corner points are obtained; outliers in the shared face features in the shared map are eliminated; and based on the normal vector of the shared face features, the visual point features of the current visual image are matched with the shared face features after elimination of the outliers to obtain shared face features having a matching relationship with the visual point features; first residual Jacobian matrices of the shared point features to the visual point features and second residual Jacobian matrices of the shared face features to the visual point features are respectively constructed according to the shared point features and the shared face features having the matching relationship; target camera poses of the current visual image are solved by using a G2O library according to the first residual Jacobian matrices and the second residual Jacobian matrices.

2. The method of claim 1, wherein, The shared map is a shared map loaded offline by a mobile robot, the mobile robot is a mobile robot carrying a camera and a laser radar sensor, and the visual robot is a mobile robot carrying a visual sensor.

3. The method of claim 1, wherein, The step of constructing the first residual Jacobian matrices of the shared point features to the visual point features according to the shared point features having the matching relationship comprises the steps of: determining second pixel coordinates of a target point in a k+1 time current visual image based on first pixel coordinates of the target point in a k time current visual image; constructing a least squares problem of the shared point features to the visual point features according to the first pixel coordinates and the second pixel coordinates; obtaining the first residual Jacobian matrices of the shared point features to the visual point features according to the least squares problem.

4. The method of claim 3, wherein, The step of constructing the second residual Jacobian matrices of the shared face features to the visual point features according to the shared face features having the matching relationship comprises the steps of: constructing a matching error of the shared face features to the visual point features based on the normal vector of the shared face features; obtaining the second residual Jacobian matrices of the shared face features to the visual point features through partial derivative processing according to the matching error.

5. The method of claim 4, wherein, The step of solving the target camera poses of the current visual image by using the G2O library according to the first residual Jacobian matrices and the second residual Jacobian matrices comprises the steps of: constructing a target loss function according to the least squares problem and the matching error; construct a total Jacobian matrix for the target loss function according to the first residual Jacobian matrix and the second residual Jacobian matrix; solve the target camera pose of the current visual image by using a G2O library according to the total Jacobian matrix; wherein the target loss function comprises matching scores of co-viewed key frames of candidate key frames in the shared map and the current visual image.

6. A multi-robot vision positioning apparatus, characterized by, comprise: an extraction module configured to extract visual point features of a current visual image of a visual robot, and shared point features and shared face features in a shared map; a matching module configured to match the visual point features of the current visual image with the shared point features and the shared face features in the shared map respectively to obtain shared point features and shared face features having a matching relationship with the visual point features; wherein the visual point features comprise extracted visual Fast corner points and visual Brief descriptors thereof, the shared point features in the shared map comprise extracted shared Fast corner points and shared Brief descriptors thereof, the matching module comprises a first matching unit and a second matching unit, the first matching unit is configured to perform feature matching by using a bag-of-words model according to visual Brief descriptors and the shared Brief descriptors, and perform angle consistency filtering processing to obtain shared Brief descriptors having a matching relationship with the visual Brief descriptors; based on a corresponding relationship between Fast corner points and Brief descriptors, shared Fast corner points having a matching relationship with the visual Fast corner points are obtained; the second matching unit is configured to eliminate outliers in shared face features in the shared map; the visual point features of the current visual image are matched with the shared face features from which the outliers are eliminated based on normal vectors of the shared face features to obtain shared face features having a matching relationship with the visual point features. a construction module configured to construct a first residual Jacobian matrix of the shared point features to the visual point features and a second residual Jacobian matrix of the shared face features to the visual point features according to the shared point features and the shared face features having the matching relationship; an estimation module configured to solve a target camera pose of the current visual image by using a G2O library according to the first residual Jacobian matrix and the second residual Jacobian matrix.

7. A storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the multi-robot visual positioning method of any one of claims 1 to 5.

8. A computer device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor executes the program to implement the multi-robot visual positioning method of any one of claims 1 to 5.

Images