Machine vision positioning method and device

Abstract

The invention discloses a machine vision positioning method, which comprises the following steps: acquiring image frame data, respectively performing global feature extraction and local feature extraction on a current image frame to respectively obtain global features and local features, selecting a feature matching mode according to a positioning state of a historical image frame adjacent to the current image frame, performing feature matching on the current image frame and each map frame in the visual map according to the selected feature matching mode to obtain a matched map frame matched with the current image frame, and determining a positioning result of the current image frame based on the matched map frame; the feature matching mode comprises a global matching mode and a prior matching mode, in the global matching mode, global feature matching is firstly carried out, then local feature matching is carried out, in the prior matching mode, positioning information of a historical image frame is utilized to obtain prior pose information of a current image frame, and the prior pose information of the current image frame is obtained; and selecting map frames in a range near the priori pose from the visual map as candidate map frames, and carrying out local feature matching.

Description

Technical Field

[0001] The present invention relates to the field of robot navigation and positioning, and in particular, to a machine vision positioning method and device. Background Art

[0002] With the widespread application of machine vision positioning technology, the requirements for the accuracy of machine vision positioning are becoming increasingly higher.

[0003] In machine vision positioning technology, one of the ways to achieve machine vision positioning is to extract feature points from image frames and combine them with odometer data such as inertial measurement unit (IMU) data to solve the positioning posture. In this positioning method, the image features extracted are usually local feature points, which leads to a lack of advantages in positioning initialization and positioning abnormality recovery during the positioning process. For example, the positioning efficiency is not fast enough and the positioning accuracy is not accurate enough. Summary of the invention

[0004] The present invention provides a machine vision positioning method to improve the advantages of positioning initialization and positioning abnormality recovery during the positioning process.

[0005] A first aspect of the present invention provides a machine vision positioning method, the method comprising:

[0006] Acquire image frame data, the image frame data including: current image frame and historical image frame,

[0007] Performing global feature extraction and local feature extraction on the current image frame respectively, obtaining global features and local features respectively, wherein the global features include: a global feature vector having a first dimension, the local features include: image position information of at least one local feature point, and a local feature vector of the local feature point, the local feature vector includes: a local feature vector having a second dimension,

[0008] According to the positioning status of the historical image frames adjacent to the current image frame, a feature matching mode is selected, and the current image frame is feature matched with each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame.

[0009] Determine the positioning result of the current image frame based on the matching map frame;

[0010] in,

[0011] The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized, and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds.

[0012] In the global matching mode, global feature matching is performed first, followed by local feature matching.

[0013] In the prior matching mode, the positioning information of the historical image frames is used to obtain the prior pose information of the current image frame, and the map frames within the vicinity of the prior pose are selected in the visual map as candidate map frames for local feature matching.

[0014] In one implementation mode, the adjacent historical image frames include: a set number of adjacent historical image frames;

[0015] In one implementation mode, the step of extracting global features and local features from the current image frame comprises:

[0016] Using the trained global feature extraction model, the global feature extraction is performed on the current image frame to obtain the global feature.

[0017] Using the trained local feature extraction model, local features are extracted from the current image frame to obtain local features.

[0018] in,

[0019] The global feature extraction model is the first deep learning neural network model.

[0020] The local feature extraction model is the second deep learning neural network model;

[0021] The dimension size of the feature vector and the numerical type of the elements depend on the structure of the deep learning neural network model.

[0022] In one implementation mode, in the global matching mode, first performing global feature matching and then performing local feature matching includes:

[0023] Match the global features of the current image frame with the global features of each map frame in the visual map,

[0024] Based on the matching results, the candidate map frames with better matching degree are selected.

[0025] For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, and the wrongly matched local feature points are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the candidate map frame.

[0026] Based on the number of matching local feature points of all candidate map frames, select the candidate map frame with the maximum number of matching local feature points,

[0027] It is determined whether the number of matching local feature points of the selected candidate map frame is greater than a set first number threshold. If so, it is determined that the global matching of the current image frame is successful, and the selected candidate map frame is a matching map frame that matches the current image frame.

[0028] In one implementation mode, the method of using the positioning information of the historical image frame to obtain the prior pose information of the current image frame includes:

[0029] Based on the a posteriori pose information of a historical image frame and the change in odometer data between the historical image frame and the current image frame, the a priori pose information of the current image frame is calculated, wherein the a priori pose information is the product of the change and the a posteriori pose.

[0030] Based on the prior pose information, map frames in the vicinity of the prior pose are selected from the visual map as candidate map frames;

[0031] In one implementation mode, the step of selecting a map frame in a vicinity of a priori posture in a visual map as a candidate map frame and performing local feature matching includes:

[0032] For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, and local feature points with incorrect matching are eliminated, and the matching local feature points with correct matching are retained to obtain the matching local feature points of the candidate map frame. Based on the number of matching local feature points of all candidate map frames, a candidate map frame with the largest number of matching local feature points is selected, and it is determined whether the number of matching local feature points of the selected candidate map frame is greater than a set second number threshold. If so, it is determined that the current image frame is successfully matched a priori, and the selected candidate map frame is the matching map frame that matches the current image frame;

[0033] or,

[0034] For each candidate map frame, the local feature points of each candidate map frame are merged to generate a local map of feature points, the local map of feature points is matched with the current image frame, the local feature points with incorrect matching are eliminated, and the matching local feature points with correct matching are retained to obtain the matching local feature points of the current image frame, and it is determined whether the number of matching local feature points of the current image frame is greater than the set third quantity threshold. If so, it is determined that the current image frame is successfully matched a priori, and the candidate map frame with the largest number of inner point matching pairs is used as the matching map frame.

[0035] In one implementation mode, the step of removing the incorrectly matched local feature points and retaining the correctly matched local feature points includes:

[0036] Using the random sampling consistency algorithm, the matching external points are eliminated to obtain the matching internal points as matching local feature points;

[0037] In one implementation mode, the a posteriori pose information of the historical image frame is the a posteriori pose information of the previous image frame adjacent to the current image frame;

[0038] The method further comprises:

[0039] Get the odometer data corresponding to the image frame data,

[0040] By fusing the image frame data and the odometer data corresponding to the image frame data, the positioning result of the current image frame is optimized.

[0041] In one implementation mode, the step of obtaining odometer data corresponding to the image frame data includes:

[0042] Get odometer data,

[0043] The image sampling time of the acquired image frame data and the odometer sampling time of the acquired odometer data are unified into the same time domain,

[0044] For each image frame sampling time in the same time domain:

[0045] If there is odometer data corresponding to the image frame sampling time, the odometer data is used as the odometer data corresponding to the image frame collected at the image frame sampling time.

[0046] If there is no odometer data corresponding to the image frame sampling time, interpolation is performed based on the odometer data collected at the odometer sampling time adjacent to the image frame sampling time, and the interpolation result is used as the odometer data corresponding to the image frame collected at the image frame sampling time;

[0047] The method of optimizing the positioning result of the current image frame by fusing the image frame data and the odometer data corresponding to the image frame data includes:

[0048] The objective function is constructed using the visual observation constraints between the image frame and its matching map frame, as well as the odometry constraints between the image frames.

[0049] The sliding window optimization method is used to solve the objective function and obtain the optimized positioning result.

[0050] In one implementation mode, the objective function is constructed by using the visual observation constraint between the image frame and its matching map frame, and the odometer constraint between the image frames, including:

[0051] Determine the bi-norm value of the reprojection error between the map feature point and the image feature point in each local feature point pair of the image frame and its matching map frame as the visual observation constraint of the local feature point pair, wherein the local feature point pair is the correctly matched matching local feature point in the image frame and its matching map frame, the matching local feature point in the matching map frame is the map feature point, and the matching local feature point in the image frame is the image feature point.

[0052] Determine the bi-norm value of the odometer data error between adjacent image frames as an odometer constraint,

[0053] Accumulate the visual observation constraints of all local feature point pairs,

[0054] Cumulatively all odometer constraints,

[0055] Sum the accumulated visual observation constraints and the accumulated odometry constraints,

[0056] Find the minimum value of the summed result as the objective function.

[0057] In one embodiment, the image frame data is from a forward-looking camera installed on the mobile robot body, and the odometer data includes inertial measurement unit data.

[0058] In one implementation mode, the method of solving the objective function by using a sliding window optimization method includes:

[0059] When the current image frame is a key frame, the current image frame is added to the sliding window. When the number of key frames contained in the sliding window is greater than the set key frame number threshold, the earliest key frame in the sliding window is deleted.

[0060] Calculate the objective function of all key frames in the current sliding window,

[0061] Solve the objective function to obtain the current image frame pose, which is the optimized pose.

[0062] A second aspect of the present application provides a machine vision positioning device, the device comprising:

[0063] An acquisition module is used to acquire image frame data, wherein the image frame data includes: a current image frame and a historical image frame.

[0064] The feature extraction module is used to extract global features and local features from the current image frame to obtain global features and local features, respectively, wherein the global features include: a global feature vector with a first dimension, the local features include: image position information of at least one local feature point, and a local feature vector of the local feature point, and the local feature vector includes: a local feature vector with a second dimension,

[0065] The feature matching module is used to select a feature matching mode according to the positioning status of the historical image frames adjacent to the current image frame, and perform feature matching on the current image frame and each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame.

[0066] A positioning module, used to determine the positioning result of the current image frame based on the matching map frame;

[0067] in,

[0068] The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized, and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds.

[0069] In the global matching mode, global feature matching is performed first, followed by local feature matching.

[0070] In the prior matching mode, the positioning information of the historical image frames is used to obtain the prior pose information of the current image frame, and the map frames within the vicinity of the prior pose are selected in the visual map as candidate map frames for local feature matching.

[0071] A third aspect of the present application provides a mobile robot, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement any of the machine vision positioning methods.

[0072] The machine vision positioning method provided in the embodiment of the present application extracts the global features and local features of the current image frame to match them with the global features and local features of the map frame in the visual map, so that the natural features in the image can be fully utilized, which is beneficial to improving the positioning accuracy. The feature matching mode is selected according to the positioning status of the historical image frame adjacent to the current image frame, which is beneficial to improving the matching efficiency, thereby improving the positioning efficiency. Compared with the traditional manually designed feature method, it has strong robustness to lighting changes and weak texture scenes, wide scene adaptability, and can realize the functions of global positioning initialization and repositioning in application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of a flow chart of a machine vision positioning method according to an embodiment of the present application.

[0074] Figure 2 A schematic diagram of a flow chart of a machine vision positioning method according to an embodiment of the present application.

[0075] Figure 3 A schematic diagram of delay calibration and data alignment.

[0076] Figure 4 A schematic diagram of image feature extraction.

[0077] Figure 5 A schematic diagram of the global matching mode.

[0078] Figure 6 A schematic diagram of a priori matching mode.

[0079] Figure 7 A schematic diagram of a process for sliding window optimization.

[0080] Figure 8 This is a schematic diagram of a graph structure that graphically describes the data relationship between map feature points, image frame postures, visual observation constraints, and reprojection constraints in this embodiment.

[0081] Fig. 9 A schematic diagram of a machine vision positioning device according to an embodiment of the present application.

[0082] Fig.10 Another schematic diagram of the machine vision positioning device according to an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0084] The embodiment of the present application performs global feature extraction and local feature extraction on the current image frame respectively, and selects a global matching mode or a priori matching mode according to the positioning status of historical image frames adjacent to the current image frame, and performs feature matching on the current image frame with each map frame in the visual map according to the selected feature matching mode. In this way, both global feature matching and local feature matching are performed, which improves the accuracy of the matched map frame that matches the current image frame, thereby helping to improve the accuracy of positioning. Due to the use of different matching modes, the matching efficiency is greatly improved, which helps to improve the efficiency of positioning.

[0085] See also Figure 1 As shown, Figure 1 A schematic diagram of a flow chart of a machine vision positioning method according to an embodiment of the present application.

[0086] The method includes:

[0087] Step 101, obtaining image frame data,

[0088] As an example, the image frame data includes: current image frame data, and historical image frame data, for example, a set number of historical image frame data adjacent to the current image frame data.

[0089] Step 102, extracting global features and local features from the current image frame to obtain global features and local features.

[0090] in,

[0091] The global feature is used to characterize the feature information of the image frame, including: a global feature vector with a first dimension,

[0092] The local feature is used to characterize the image position information of the feature point in the image frame and the feature information of the image space mapped by the feature point, including: the image position information of at least one local feature point and the local feature vector of the local feature point, the local feature vector including: a local feature vector with a second dimension, since there are usually multiple local feature points, the local feature includes multiple local feature vectors, and each local feature point corresponds to its local feature vector;

[0093] The first dimension and the second dimension may be the same or different.

[0094] As an example, the trained global feature extraction model is used to extract global features of the current image frame to obtain global features.

[0095] Using the trained local feature extraction model, local features are extracted from the current image frame to obtain local features.

[0096] in,

[0097] The global feature extraction model is the first deep learning neural network model.

[0098] The local feature extraction model is the second deep learning neural network model;

[0099] The dimension size of the feature vector and the numerical type of the elements depend on the structure of the deep learning neural network model.

[0100] Step 103, according to the positioning state of the historical image frame adjacent to the current image frame, a feature matching mode is selected, and the current image frame is feature matched with each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame.

[0101] Furthermore, a pair of local feature points matching the local feature points in the current image frame and the local feature points in the matching map frame can be obtained.

[0102] in,

[0103] Each map frame in the visual map is subjected to global feature extraction and local feature extraction respectively, and global features and local features are obtained respectively as map features of the map frame. The extraction method may be the same as that of the image frame.

[0104] The local feature point pair includes: the matching local feature point located in the image frame and the matching local feature point located in the matching map frame. For the convenience of writing, in the local feature point pair, the matching local feature point located in the image frame is called the image feature point, and the matching local feature point located in the matching map frame is called the map feature point.

[0105] The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized (e.g., booting or restarting), and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds.

[0106] In the global matching mode, global feature matching is performed first. Since the global feature is a global feature vector with a first dimension, it is equivalent to a coarse matching. Then local feature matching is performed. Since the local feature includes the local feature vectors of each local feature point, it is equivalent to a fine matching.

[0107] As an example, in the global matching mode, feature matching of the current image frame with each map frame in the visual map includes:

[0108] Match the global features of the current image frame with the global features of each map frame in the visual map,

[0109] Based on the matching results, the candidate map frames with better matching degree are selected.

[0110] For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, and the wrongly matched local feature points are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the candidate map frame.

[0111] Based on the number of matching local feature points of all candidate map frames, select the candidate map frame with the maximum number of matching local feature points,

[0112] It is determined whether the number of matching local feature points of the selected candidate map frame is greater than a set first number threshold. If so, it is determined that the global matching of the current image frame is successful, and the selected candidate map frame is a matching map frame that matches the current image frame.

[0113] In the prior matching mode, the positioning information of the historical image frames, that is, the posterior pose information of the historical image frames, is used to obtain the prior pose information of the current image frame, and the map frames within the vicinity of the prior pose are selected in the visual map as candidate map frames. This is conducive to quickly locking the candidate map frames and performing local feature matching between the current image frame and the candidate map frames, which is equivalent to performing precise matching.

[0114] As an example, in the prior matching mode, feature matching of the current image frame with each map frame in the visual map includes:

[0115] Based on the a posteriori pose information of a historical image frame and the change in odometer data between the historical image frame and the current image frame, the a priori pose information of the current image frame is calculated, wherein the a priori pose information is the product of the change and the a posteriori pose.

[0116] Based on the prior pose information, map frames in the vicinity of the prior pose are selected from the visual map as candidate map frames.

[0117] For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, and the wrongly matched local feature points are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the candidate map frame.

[0118] Based on the number of matching local feature points of all candidate map frames, select the candidate map frame with the maximum number of matching local feature points,

[0119] It is determined whether the number of matching local feature points of the selected candidate map frame is greater than a set second number threshold. If so, it is determined that the current image frame is successfully matched a priori, and the selected candidate map frame is a matching map frame that matches the current image frame.

[0120] or,

[0121] The local feature points of each candidate map frame are merged to generate a local feature point map, and the local feature point map is matched with the current image frame. The wrongly matched local feature points are removed, and the correctly matched local feature points are retained to obtain the matched local feature points of the current image frame.

[0122] It is determined whether the number of matching local feature points of the current image frame is greater than a set third number threshold. If so, it is determined that the current image frame is successfully matched a priori, and the candidate map frame with the maximum number of inlier matching pairs is used as the matching map frame.

[0123] Step 104: Determine the positioning result of the current image frame based on the matching map frame.

[0124] Since the matching map frame pose can be obtained when the visual map is constructed, the positioning result of the current image frame can be obtained based on the matching map frame pose.

[0125] Since the current image frame pose determined by matching the map frame is the result of visual matching constraints, when the current image frame has rich features or a small number of matching local feature points, the accuracy of the current image frame pose is poor, so the current image frame pose can be further optimized through step 105.

[0126] Step 105, obtaining the odometer data corresponding to the image frame data, and optimizing the positioning result of the current image frame by fusing the image frame data and the odometer data corresponding to the image frame data.

[0127] As an example, the objective function is constructed by using the visual observation constraints between the image frame and its matching map frame, as well as the odometer constraints between the image frames. The sliding window optimization method is used to solve the objective function to obtain the optimized positioning result.

[0128] in,

[0129] Odometer data includes inertial measurement unit data,

[0130] Visual observation constraints include: the reprojection error between the map feature point and the image feature point in any local feature point pair between the image frame and its matching map frame,

[0131] Odometry constraints include: odometry data error between adjacent image frames,

[0132] The objective function is the minimum value of the sum of the visual observation constraints of all local feature point pairs in the image frame and the sum of all odometry constraints.

[0133] The embodiment of the present application performs both local feature matching and global feature matching on the image frame, which enables fast and accurate positioning in cases of positioning initialization and positioning anomalies, and can realize the functions of global positioning initialization and repositioning in application scenarios; since the image frame includes natural features in the environment, it is not dependent on artificial positioning marks, and is easier to implement and use. The application of global features and local extraction networks based on deep learning has stronger robustness to lighting changes and weak texture scenes than traditional methods of artificially designed features.

[0134] To facilitate understanding of the embodiments of the present application, the following is an example of an image frame captured by a forward-looking camera and odometer data. It should be understood that the embodiments of the present application are not limited to image frames and odometer data captured by a forward-looking camera, and may also be applicable to image frames and other inertial measurement unit data captured by a downward-looking camera.

[0135] What should be understood is that the difference between visual positioning based on forward-looking cameras and downward-looking cameras is reflected in the following: although the image frames captured by the forward-looking cameras are affected by lighting to a certain extent, because the camera adopts a forward-looking layout, the effective field of view is large, and more abundant natural features can be observed. The image information is rich, the distinction is good, the scene adaptability is wide, and it is more suitable for weak-texture scenes. This also means that positioning using natural features in image frames does not rely on artificial markers added to the environment, including but not limited to QR codes, ribbons, etc., to facilitate implementation and deployment.

[0136] See also Figure 2 As shown, Figure 2 A schematic diagram of a flow chart of a machine vision positioning method according to an embodiment of the present application.

[0137] The method includes:

[0138] Step 201, obtain an image frame from a front-view camera and odometer data from an odometer, align the image frame data with the odometer data so that each image frame data corresponds to the odometer data,

[0139] As an example, the odometer is a wheel encoder,

[0140] Generally speaking, the clock sources of the camera and the odometer are inconsistent. In addition, the sampling frequencies are not consistent, and the odometer sampling frequency is usually higher than the camera sampling frequency. The data of the two need to be synchronized to facilitate data fusion.

[0141] Data synchronization usually includes: delay calibration and data alignment.

[0142] Delay calibration is to calculate the relative delay between the clocks of the camera and the odometer. Through delay calibration, the camera sampling time domain and the odometer sampling domain are unified into the same time domain, that is, the camera sampling timestamp information and the odometer sampling timestamp information are unified into the same timestamp system. If the timestamp synchronization of the camera and the odometer is achieved at the hardware or software level, the delay calibration process can be omitted.

[0143] Data alignment aligns the data of the synchronous camera and odometer according to the delay of the delay calibration to obtain the odometer data corresponding to the camera sampling time. In this way, each image frame data can correspond to its odometer data.

[0144] See also Figure 3 As shown, Figure 3A schematic diagram of delay calibration and data alignment. The camera sampling timestamp information and the odometer sampling timestamp information are unified into the same time domain. In the figure, the hollow circle represents the image frame data and the solid circle represents the odometer data. Since the odometer sampling frequency is higher than the camera sampling frequency, the number of odometer sampling points is greater than the number of camera sampling points in the same time.

[0145] When performing data alignment, the odometer data corresponding to the image frame data is matched according to the calibrated delay. As an example, for each image frame sampling time:

[0146] If there is odometer data corresponding to the image frame sampling time, the odometer data is used as the odometer data corresponding to the image frame collected at the image frame sampling time.

[0147] If there is no corresponding odometer data to match the image frame, that is, there is no corresponding odometer data at the sampling time of the image frame, then interpolation can be performed based on the odometer data collected at the odometer sampling time adjacent to the image frame sampling time, and the interpolated odometer data can be used as the odometer data corresponding to the image frame collected at the image frame sampling time.

[0148] Similarly, for the current image frame, its corresponding odometer data can also be obtained in a data synchronization manner.

[0149] Step 202, using the trained global feature extraction network model, performing global feature extraction on the current image frame to reduce the dimension and compress the image frame data to obtain global features, and using the trained local feature extraction network model, performing local feature extraction on the current image frame to obtain local feature point information in the image frame to obtain local features,

[0150] As an example, the global feature extraction network model is a first deep learning neural network model, and the local feature extraction network model is a second deep learning neural network model.

[0151] After the global feature extraction network model performs forward reasoning on the input current image frame, it can output a D g dimensional global descriptor vector The numerical type of the elements of the descriptor vector output by the global feature extraction network model, and the dimension D of the global feature vector descriptor g , both of which can be modified by adjusting the structure of the global feature extraction network. The numerical type of the element can be binary or floating point.

[0152] After the local feature extraction network model performs forward reasoning on the input current image frame, it can output the image position information of N local feature points, such as pixel coordinates p i , and each local feature point pi The corresponding D l dimensional descriptor vector To characterize the image space feature information mapped by the local feature points, where i = 1, 2, 3, ..., N, N is a natural number not less than 1. The number of local feature points N can be set by inputting the configuration parameters of the local feature extraction network model. The dimension D of the local feature point descriptor vector l The numerical types of the and elements are the same as the global feature extraction network model and can also be modified by adjusting the network structure.

[0153] The global features and local features based on deep learning have certain scale invariance. If we want to further improve the robustness of global features and local features to image scale, we can use the image pyramid method to extract image feature information at different scales.

[0154] The trained global feature extraction network model and local feature extraction network model can be deployed in the mobile robot through transplantation and quantization.

[0155] It should be understood that the training of the global feature extraction network model and the local feature extraction network model can be obtained by training with sample image data collected by the forward-looking camera.

[0156] See also Figure 4 As shown, Figure 4 A schematic diagram of image feature extraction. In the figure, the global feature is D g dimensional vector, a local feature is D for each feature point l dimensional vector, when the number of feature points is N, the local features include N different D l dimensional vector.

[0157] Step 203 , matching the extracted global features and local features with the map features of the map frame in the visual map, so as to establish a one-to-one correspondence between the image features of the image frame and the map features in the visual map.

[0158] For example, when the features of an image frame match those of a map frame, the local feature points of the image frame and the map feature points in the correct matching pair are imaging observations of the same spatial point in the physical space. In this way, accurate constraints can be formed on the pose of the image frame through the camera projection model; conversely, an incorrect matching pair will constitute incorrect constraints on the pose of the image frame, thereby affecting the accuracy of the image frame pose solution and even causing errors in the image frame pose solution.

[0159] In this embodiment, different feature matching methods are used based on the positioning status of a set number of historical image frames adjacent to the current image frame. Specifically, during the initialization of positioning and / or after positioning failure occurs during the positioning process, since there is no more accurate estimate of the prior pose of the current image frame, a global matching mode is used for feature matching; when the positioning and matching of adjacent historical image frames is successful, since there is a more accurate estimate of the pose of the historical image frame, the pose of the historical image frame and the odometer data output by the odometer can be used to make a better estimate of the prior pose of the current image frame, and at this time, the prior pose is used as an auxiliary prior matching mode.

[0160] As an example, see Figure 5 As shown, Figure 5 This is a schematic diagram of the global matching mode. The global matching mode is as follows:

[0161] First, the global descriptor vector of the current image frame is matched with the global descriptor vector of the map frame in the visual map for global feature matching to perform rough matching. Based on the global descriptor matching results, map frames with better matching degrees are selected as candidate map frames, for example, K candidate map frames with the highest matching scores, where the matching scores can be calculated using cosine distance (floating-point descriptor vector) or Hamming distance (binary descriptor vector), and K can be selected according to actual conditions;

[0162] Then, all K candidate map frames are traversed, and the local features of the current image frame are matched with the local features of the candidate map frames to perform frame-by-frame point pair precise matching, that is, the 2D point pair of the current image frame is precisely matched with the 3D point pair of the candidate map frame, and the unmatched external points are eliminated based on the random sampling consensus (RANSAC) method, and the matching pairs of internal points that match each candidate map frame are recorded to obtain local feature point pairs;

[0163] Finally, based on the inlier matching pairs that match each candidate map frame, the candidate map frame with the largest number of inlier matching pairs is selected, and it is determined whether the number of inlier matching pairs in the selected candidate map frame is greater than a first quantity threshold, such as the effective inlier quantity threshold N. inlier_t , if it is greater than , then the global matching and positioning of the current image frame is considered to be successful, and the selected candidate map frame is used as the matching map frame. Otherwise, the global matching and positioning of the current image frame is considered to have failed.

[0164] As an example, see Figure 6 As shown, Figure 6 A schematic diagram of a priori matching mode is shown below.

[0165] Given that the premise for selecting this mode is that the positioning and matching of the adjacent historical image frames of the current image frame is successful, the current image frame I jThe adjacent historical image frame, for example, the previous image frame I j-1 After reliable pose estimation, we can obtain the posterior pose information with high confidence. The corresponding odometer increment between two image frames is ΔT j,j-1 , then the current image frame I is obtained j The prior pose of is the product of the odometer increment and the posterior pose, which can be expressed mathematically as:

[0166]

[0167] Unlike the global matching mode that filters candidate map frames through global descriptors, the prior matching mode uses the current prior pose information Selecting map frames in the vicinity of the prior pose in the visual map as candidate map frames;

[0168] After obtaining each candidate map frame, one of the ways is to use the same method as traversing each candidate map frame in the global matching mode to screen out matching map frames and local feature point pairs, and determine whether the number of inner point matching pairs in the selected candidate map frame is greater than a second quantity threshold. If so, it is considered that the global matching positioning of the current image frame is successful, and the selected candidate map frame is used as the matching map frame. Otherwise, it is considered that the global matching positioning of the current image frame fails.

[0169] The second method is that since each candidate map frame is adjacent and continuous in the map space in the prior matching mode, all map feature points associated with each candidate map frame can also be merged to generate a local map of feature points. Then, the local map of feature points is matched with the current image frame for local features, and then the random sampling consensus (RANSAC) method is used to eliminate mismatched external points to obtain inlier matching pairs of the feature points of the current image frame and the map feature points as local feature point pairs. It is determined whether the number of inlier matching pairs of the current image frame is greater than a third quantity threshold. If so, it is considered that the prior matching and positioning of the current image frame is successful, and the candidate map frame with the largest number of inlier matching pairs is used as the matching map frame. Otherwise, it is considered that the prior matching and positioning of the current image frame has failed.

[0170] Step 204, determining the pose of the current image frame according to the pose information of the matching map frame, and optimizing the pose.

[0171] Based on the pose information of the matching map frame, the pose of the current image frame is obtained through visual multi-point PnP solution.

[0172] The pose obtained by the prior matching mode or the global matching mode is completely constrained by the visual matching, which is affected by the current feature richness and the number of feature matches. If there are many mismatches, it is easy to cause positioning failure. In order to obtain a more stable, accurate and smoother positioning result, the odometer constraint and the visual observation constraint can be fused to jointly optimize the robot pose.

[0173] This embodiment adopts a sliding window optimization method, retains a certain amount of key frame information, and uses multiple historical key frames to constrain the positioning result of the current image frame. In this way, even if there are not enough matching feature points, relatively good positioning results can still be obtained, while also effectively balancing computational efficiency and positioning performance.

[0174] See also Figure 7 As shown, Figure 7 A flowchart for sliding window optimization. Keyframes are screened by motion distance, but both keyframes and non-keyframes will be optimized. The difference is that non-keyframes will be deleted after one optimization, while keyframes will remain in the sliding window until they overflow the sliding window. When the image frame to be optimized enters the sliding window, because the number of keyframes contained in the sliding window is fixed, that is, the sliding window length is fixed, the earliest keyframe in the sliding window needs to be deleted to realize the sliding of the window, and the image frame information in the sliding window is used to optimize the posture of the image frame to be optimized.

[0175] The graph optimization problem that needs to be solved for sliding window optimization. Figure 8 As shown, Figure 8 This is a diagram of a graph structure that graphically describes the data relationship between map feature points, image frame poses, visual observation constraints, and reprojection constraints. In the figure, each dot represents a node. The yellow dot represents the matched map feature point, whose pose has been obtained during map construction and remains unchanged during sliding window optimization. The blue dot represents the pose of the image frame, which is obtained by visual multi-point PnP solution after matching. The pose of the image frame is the initial value. Each square represents an edge constraint. The yellow square represents the visual observation constraint of a local feature point pair, specifically, the reprojection constraint r between the map feature point and the image feature point in the local feature point pair. v , which can be obtained in the feature matching process; the green square represents the odometer constraint r between the two image frames. o , is the pose increment between two image frames obtained from the odometer data. In the sliding window optimization process, the odometer constraints between image frames and the visual observation constraints of each image frame are integrated to obtain the final pose estimate.

[0176] In the whole sliding window optimization process, because the position of the map feature points is fixed, there is no direct visual constraint between the image frames. However, because there is also an odometer constraint between the image frames, the visual constraint of each image frame indirectly constitutes a weak constraint on the position and posture of other image frames. In this embodiment, the odometer data is simply used as measurement data, and the state or parameters of the odometer are not involved in the optimization.

[0177] The objective function of the sliding window graph optimization is: the minimum value of the sum of the visual observation constraints of all local feature point pairs in the image frame and the sum of all odometer constraints. The visual observation constraints and odometer constraints are calculated using the two-norm value, which can be expressed mathematically as follows:

[0178] min{∑‖r o ‖ 2 +∑‖r v ‖ 2}

[0179] in,‖‖ 2 represents 2-normal calculation, r o is the odometry constraint between adjacent key frames in the sliding window, r v is the reprojection constraint between the map feature point and the image feature point in a local feature point in the sliding window, ∑ represents the sum of the constraints on all edges in the sliding window, that is, ∑‖r o ‖ 2 represents the accumulation of all odometer constraints, ∑‖r v ‖ 2 Represents the accumulation of all reprojection constraints.

[0180] The optimized posture is obtained by solving the current image frame posture when the objective function is minimized. Specifically, when the current image frame is a key frame, the current image frame is added to the sliding window. When the number of key frames contained in the sliding window is greater than the set key frame number threshold, the earliest key frame in the sliding window is deleted, and the objective function of all key frames in the current sliding window is calculated to obtain the current image frame posture when the objective function is minimized.

[0181] The embodiment of the present application performs global feature extraction and local feature extraction on the current image frame from the forward-looking camera, and selects a feature matching mode according to the positioning status of the historical image frame, which is beneficial to improving both the accuracy and efficiency of positioning.

[0182] See also Fig. 9 As shown, Fig. 9 This is a schematic diagram of a machine vision positioning device according to an embodiment of the present application. The device includes:

[0183] The acquisition module is used to obtain image frame data.

[0184] The feature extraction module is used to extract global features and local features of the current image frame to obtain global features and local features respectively.

[0185] The feature matching module is used to select a feature matching mode according to the positioning status of the historical image frames adjacent to the current image frame, and perform feature matching on the current image frame and each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame.

[0186] The positioning module is used to determine the positioning result of the current image frame based on the matching map frame.

[0187] As an example,

[0188] The acquisition module is also used to acquire odometer data corresponding to the image frame data.

[0189] The feature matching module is also used to determine the prior pose information of the current image frame according to the IMU change and the a posteriori pose in the prior matching mode, so as to quickly select the candidate map frame from the visual map.

[0190] The positioning module is also configured to optimize the positioning result of the current image frame by fusing the image frame data and the odometer data corresponding to the image frame data.

[0191] See also Fig.10 As shown, Fig.10 Another schematic diagram of a machine vision positioning device according to an embodiment of the present application. The device comprises: a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the steps of the machine vision positioning method according to the embodiment of the present application.

[0192] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0193] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0194] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the machine vision positioning method described in the embodiment of the present application are implemented.

[0195] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0196] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0197] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A machine vision positioning method, characterized in that: The method includes: Acquire image frame data, the image frame data including: current image frame and historical image frame, Performing global feature extraction and local feature extraction on the current image frame respectively, obtaining global features and local features respectively, wherein the global features include: a global feature vector having a first dimension, the local features include: image position information of at least one local feature point, and a local feature vector of the local feature point, the local feature vector includes: a local feature vector having a second dimension, According to the positioning status of the historical image frames adjacent to the current image frame, a feature matching mode is selected, and the current image frame is feature matched with each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame. Determine the positioning result of the current image frame based on the matching map frame; in, The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized, and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds. In the global matching mode, global feature matching is performed first, followed by local feature matching. In the prior matching mode, the positioning information of the historical image frames is used to obtain the prior pose information of the current image frame, and the map frames within the vicinity of the prior pose are selected in the visual map as candidate map frames for local feature matching.

2. The machine vision positioning method according to claim 1, characterized in that: The adjacent historical image frames include: a set number of adjacent historical image frames; The extracting global features and local features of the current image frame respectively includes: Using the trained global feature extraction model, the global feature extraction is performed on the current image frame to obtain the global feature. Using the trained local feature extraction model, local features are extracted from the current image frame to obtain local features. in, The global feature extraction model is the first deep learning neural network model. The local feature extraction model is the second deep learning neural network model; The dimension size of the feature vector and the numerical type of the elements depend on the structure of the deep learning neural network model.

3. The machine vision positioning method according to claim 2, characterized in that: In the global matching mode, global feature matching is performed first, and then local feature matching is performed, including: Match the global features of the current image frame with the global features of each map frame in the visual map, Based on the matching results, the candidate map frames with better matching degree are selected. For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, and the wrongly matched local feature points are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the candidate map frame. Based on the number of matching local feature points of all candidate map frames, select the candidate map frame with the maximum number of matching local feature points, It is determined whether the number of matching local feature points of the selected candidate map frame is greater than a set first number threshold. If so, it is determined that the global matching of the current image frame is successful, and the selected candidate map frame is a matching map frame that matches the current image frame.

4. The machine vision positioning method according to any one of claims 1 to 3, characterized in that: The method of using the positioning information of the historical image frames to obtain the priori posture information of the current image frame includes: Based on the a posteriori pose information of a historical image frame and the change in odometer data between the historical image frame and the current image frame, the a priori pose information of the current image frame is calculated, wherein the a priori pose information is the product of the change and the a posteriori pose. Based on the prior pose information, map frames in the vicinity of the prior pose are selected from the visual map as candidate map frames; The selecting of map frames in the vicinity of the prior pose in the visual map as candidate map frames for local feature matching includes: For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, and local feature points with incorrect matching are eliminated, and the matching local feature points with correct matching are retained to obtain the matching local feature points of the candidate map frame. Based on the number of matching local feature points of all candidate map frames, a candidate map frame with the largest number of matching local feature points is selected, and it is determined whether the number of matching local feature points of the selected candidate map frame is greater than a set second number threshold. If so, it is determined that the current image frame is successfully matched a priori, and the selected candidate map frame is the matching map frame that matches the current image frame; or, For each candidate map frame, the local feature points of each candidate map frame are merged to generate a local map of feature points, the local map of feature points is matched with the current image frame, the local feature points with incorrect matching are eliminated, and the matching local feature points with correct matching are retained to obtain the matching local feature points of the current image frame, and it is determined whether the number of matching local feature points of the current image frame is greater than the set third quantity threshold. If so, it is determined that the current image frame is successfully matched a priori, and the candidate map frame with the largest number of inner point matching pairs is used as the matching map frame.

5. The machine vision positioning method according to claim 4, characterized in that: The method of removing the incorrectly matched local feature points and retaining the correctly matched local feature points includes: Using the random sampling consistency algorithm, the matching external points are eliminated to obtain the matching internal points as matching local feature points; The a posteriori pose information of the historical image frame is the a posteriori pose information of the previous image frame adjacent to the current image frame; The method further comprises: Get the odometer data corresponding to the image frame data, By fusing the image frame data and the odometer data corresponding to the image frame data, the positioning result of the current image frame is optimized.

6. The machine vision positioning method according to claim 5, characterized in that: The obtaining of odometer data corresponding to the image frame data includes: Get odometer data, The image sampling time of the acquired image frame data and the odometer sampling time of the acquired odometer data are unified into the same time domain, For each image frame sampling time in the same time domain: If there is odometer data corresponding to the image frame sampling time, the odometer data is used as the odometer data corresponding to the image frame collected at the image frame sampling time. If there is no odometer data corresponding to the image frame sampling time, interpolation is performed based on the odometer data collected at the odometer sampling time adjacent to the image frame sampling time, and the interpolation result is used as the odometer data corresponding to the image frame collected at the image frame sampling time; The method of optimizing the positioning result of the current image frame by fusing the image frame data and the odometer data corresponding to the image frame data includes: The objective function is constructed using the visual observation constraints between the image frame and its matching map frame, as well as the odometry constraints between the image frames. The sliding window optimization method is used to solve the objective function and obtain the optimized positioning result.

7. The machine vision positioning method according to claim 6, characterized in that: The objective function is constructed by using the visual observation constraint between the image frame and its matching map frame, and the odometer constraint between the image frames, including: Determine the bi-norm value of the reprojection error between the map feature point and the image feature point in each local feature point pair of the image frame and its matching map frame as the visual observation constraint of the local feature point pair, wherein the local feature point pair is the correctly matched local feature point in the image frame and its matching map frame, the matched local feature point in the matching map frame is the map feature point, and the matched local feature point in the image frame is the image feature point. Determine the bi-norm value of the odometer data error between adjacent image frames as an odometer constraint, Accumulate the visual observation constraints of all local feature point pairs, Cumulatively all odometer constraints, Sum the accumulated visual observation constraints and the accumulated odometry constraints, Find the minimum value of the summed result as the objective function.

8. The machine vision positioning method according to claim 7, characterized in that: The image frame data comes from a forward-looking camera installed on the mobile robot body, and the odometer data includes inertial measurement unit data. The sliding window optimization method is used to solve the objective function, including: When the current image frame is a key frame, the current image frame is added to the sliding window. When the number of key frames contained in the sliding window is greater than the set key frame number threshold, the earliest key frame in the sliding window is deleted. Calculate the objective function of all key frames in the current sliding window, Solve the objective function to obtain the current image frame pose, which is the optimized pose.

9. A machine vision positioning device, characterized in that: The device includes: An acquisition module is used to acquire image frame data, wherein the image frame data includes: a current image frame and a historical image frame. The feature extraction module is used to extract global features and local features from the current image frame to obtain global features and local features, respectively, wherein the global features include: a global feature vector with a first dimension, the local features include: image position information of at least one local feature point, and a local feature vector of the local feature point, and the local feature vector includes: a local feature vector with a second dimension, The feature matching module is used to select a feature matching mode according to the positioning status of the historical image frames adjacent to the current image frame, and perform feature matching on the current image frame and each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame. A positioning module, used to determine the positioning result of the current image frame based on the matching map frame; in, The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized, and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds. In the global matching mode, global feature matching is performed first, followed by local feature matching. In the prior matching mode, the positioning information of the historical image frames is used to obtain the prior pose information of the current image frame, and the map frames within the vicinity of the prior pose are selected in the visual map as candidate map frames for local feature matching.

10. A mobile robot, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the machine vision positioning method according to any one of claims 1 to 8.

Images