A machine vision positioning method and device

Abstract

The present application discloses a machine vision positioning method, including: acquiring image frame data, performing global feature extraction and local feature extraction on the current image frame respectively to obtain global features and local features respectively, selecting a feature matching mode according to the positioning status of historical image frames adjacent to the current image frame, performing feature matching on the current image frame with each map frame in a visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame, and determining the positioning result of the current image frame based on the matching map frame; the feature matching mode includes: a global matching mode and a priori matching mode, in the global matching mode, global feature matching is performed first, and then local feature matching is performed, in the priori matching mode, the positioning information of the historical image frames is used to obtain the priori posture information of the current image frame, and map frames within a range near the prior posture are selected in the visual map as candidate map frames to perform local feature matching.

Description

Technical Field

[0001] The present invention relates to the field of robot navigation and positioning, and in particular, to a machine vision positioning method and device. Background Art

[0002] With the widespread application of machine vision positioning technology, the requirements for the accuracy of machine vision positioning are becoming increasingly higher.

[0003] In machine vision positioning technology, one way to achieve machine vision positioning is to extract feature points from image frames and combine them with odometry data such as inertial measurement unit (IMU) data to solve the positioning pose. In this positioning method, the image features extracted are usually local feature points, which leads to a lack of advantages in positioning initialization and positioning anomaly recovery during the positioning process. For example, the positioning efficiency is not fast enough and the positioning accuracy is not accurate enough. Summary of the Invention

[0004] The present invention provides a machine vision positioning method to improve the advantages of positioning initialization and positioning abnormality recovery during the positioning process.

[0005] A first aspect of the present invention provides a machine vision positioning method, the method comprising:

[0006] Acquire image frame data, the image frame data including: current image frame and historical image frames,

[0007] Performing global feature extraction and local feature extraction on the current image frame to obtain global features and local features, respectively, wherein the global feature includes: a global feature vector having a first dimension, and the local feature includes: image position information of at least one local feature point and a local feature vector of the local feature point, and the local feature vector includes: a local feature vector having a second dimension,

[0008] According to the positioning status of the historical image frames adjacent to the current image frame, a feature matching mode is selected. According to the selected feature matching mode, the current image frame is feature matched with each map frame in the visual map to obtain a matching map frame that matches the current image frame.

[0009] Determine the positioning result of the current image frame based on the matching map frame;

[0010] in,

[0011] The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized, and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds.

[0012] In global matching mode, global feature matching is performed first, followed by local feature matching.

[0013] In the prior matching mode, the positioning information of the historical image frames is used to obtain the prior pose information of the current image frame, and the map frames within the range near the prior pose are selected in the visual map as candidate map frames for local feature matching.

[0014] In one embodiment, the adjacent historical image frames include: a set number of adjacent historical image frames;

[0015] In one embodiment, the step of extracting global features and local features from the current image frame includes:

[0016] Using the trained global feature extraction model, global feature extraction is performed on the current image frame to obtain global features.

[0017] Using the trained local feature extraction model, local features are extracted from the current image frame to obtain local features.

[0018] in,

[0019] The global feature extraction model is the first deep learning neural network model,

[0020] The local feature extraction model is the second deep learning neural network model;

[0021] The dimension of the feature vector and the numerical type of the elements depend on the structure of the deep learning neural network model.

[0022] In one embodiment, in the global matching mode, global feature matching is performed first and then local feature matching is performed, including:

[0023] Match the global features of the current image frame with the global features of each map frame in the visual map,

[0024] Based on the matching results, the candidate map frames with better matching degree are selected.

[0025] For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, the incorrectly matched local feature points are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the candidate map frame.

[0026] Based on the number of matching local feature points of all candidate map frames, the candidate map frame with the largest number of matching local feature points is selected.

[0027] Determine whether the number of matching local feature points of the selected candidate map frame is greater than a set first number threshold. If so, determine that the current image frame is globally matched successfully, and the selected candidate map frame is the matching map frame that matches the current image frame.

[0028] In one embodiment, the method of using the positioning information of the historical image frames to obtain the prior pose information of the current image frame includes:

[0029] Based on the posterior pose information of a historical image frame and the change in odometer data between the historical image frame and the current image frame, the prior pose information of the current image frame is calculated, wherein the prior pose information is the product of the change and the posterior pose.

[0030] Based on the prior pose information, map frames within a range near the prior pose are selected from the visual map as candidate map frames;

[0031] In one embodiment, the step of selecting a map frame within a range near a priori pose in the visual map as a candidate map frame and performing local feature matching includes:

[0032] For each candidate map frame, perform local feature matching between the candidate map frame and the current image frame, remove incorrectly matched local feature points, retain correctly matched local feature points, and obtain matching local feature points of the candidate map frame; based on the number of matching local feature points of all candidate map frames, select the candidate map frame with the largest number of matching local feature points; determine whether the number of matching local feature points of the selected candidate map frame is greater than a set second number threshold; if so, determine that the priori matching of the current image frame is successful, and the selected candidate map frame is the matching map frame that matches the current image frame;

[0033] or,

[0034] For each candidate map frame, the local feature points of each candidate map frame are merged to generate a local map of feature points. The local map of feature points is matched with the current image frame, the local feature points with incorrect matching are eliminated, and the matching local feature points with correct matching are retained to obtain the matching local feature points of the current image frame. It is determined whether the number of matching local feature points of the current image frame is greater than the set third quantity threshold. If so, it is determined that the prior matching of the current image frame is successful, and the candidate map frame with the largest number of inlier matching pairs is used as the matching map frame.

[0035] In one embodiment, the step of removing incorrectly matched local feature points and retaining correctly matched local feature points includes:

[0036] Using the random sampling consistency algorithm, the matching external points are eliminated to obtain the matching internal points as matching local feature points;

[0037] In one embodiment, the a posteriori pose information of the historical image frame is the a posteriori pose information of the previous image frame adjacent to the current image frame;

[0038] The method further comprises:

[0039] Get the odometer data corresponding to the image frame data,

[0040] By fusing the image frame data and the odometer data corresponding to the image frame data, the positioning result of the current image frame is optimized.

[0041] In one embodiment, obtaining odometer data corresponding to the image frame data includes:

[0042] Get odometer data,

[0043] Unify the image sampling time of the acquired image frame data and the odometer sampling time of the acquired odometer data into the same time domain,

[0044] For each image frame sampling time in the same time domain:

[0045] If there is odometer data corresponding to the image frame sampling time, the odometer data is used as the odometer data corresponding to the image frame collected at the image frame sampling time.

[0046] If there is no odometer data corresponding to the image frame sampling time, interpolation is performed based on the odometer data collected at the odometer sampling time adjacent to the image frame sampling time, and the interpolation result is used as the odometer data corresponding to the image frame collected at the image frame sampling time;

[0047] The optimizing the positioning result of the current image frame by fusing the image frame data and the odometer data corresponding to the image frame data includes:

[0048] The objective function is constructed using the visual observation constraints between the image frame and its matching map frame, as well as the odometry constraints between the image frames.

[0049] The sliding window optimization method is used to solve the objective function and obtain the optimized positioning result.

[0050] In one embodiment, constructing the objective function using the visual observation constraint between the image frame and its matching map frame, and the odometry constraint between the image frames, includes:

[0051] Determine the bi-norm value of the reprojection error between the map feature point and the image feature point in each local feature point pair of the image frame and its matching map frame as the visual observation constraint of the local feature point pair, wherein the local feature point pair is the correctly matched matching local feature point in the image frame and its matching map frame, the matching local feature point in the matching map frame is the map feature point, and the matching local feature point in the image frame is the image feature point.

[0052] Determine the bi-norm value of the odometry data error between adjacent image frames as an odometry constraint,

[0053] Accumulate the visual observation constraints of all local feature point pairs,

[0054] Accumulate all odometry constraints,

[0055] Sum the accumulated visual observation constraints and the accumulated odometry constraints,

[0056] Find the minimum value of the summed result as the objective function.

[0057] In one embodiment, the image frame data comes from a forward-looking camera installed on the mobile robot body, and the odometer data includes inertial measurement unit data.

[0058] In one embodiment, the method of solving the objective function by using a sliding window optimization method includes:

[0059] When the current image frame is a key frame, the current image frame is added to the sliding window. When the number of key frames contained in the sliding window is greater than the set key frame number threshold, the earliest key frame in the sliding window is deleted.

[0060] Calculate the objective function of all key frames in the current sliding window,

[0061] Solve the objective function to obtain the current image frame pose, which is the optimized pose.

[0062] A second aspect of the present application provides a machine vision positioning device, the device comprising:

[0063] An acquisition module is used to acquire image frame data, wherein the image frame data includes: a current image frame and a historical image frame.

[0064] The feature extraction module is used to perform global feature extraction and local feature extraction on the current image frame to obtain global features and local features respectively, wherein the global feature includes: a global feature vector with a first dimension, and the local feature includes: image position information of at least one local feature point and a local feature vector of the local feature point, and the local feature vector includes: a local feature vector with a second dimension,

[0065] The feature matching module is used to select a feature matching mode based on the positioning status of the historical image frames adjacent to the current image frame, and perform feature matching on the current image frame with each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame.

[0066] A positioning module, configured to determine a positioning result of a current image frame based on a matching map frame;

[0067] in,

[0068] The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized, and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds.

[0069] In global matching mode, global feature matching is performed first, followed by local feature matching.

[0070] In the prior matching mode, the positioning information of the historical image frames is used to obtain the prior pose information of the current image frame, and the map frames within the range near the prior pose are selected in the visual map as candidate map frames for local feature matching.

[0071] A third aspect of the present application provides a mobile robot, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement any of the machine vision positioning methods.

[0072] The machine vision positioning method provided in the embodiment of the present application extracts the global features and local features of the current image frame to match them with the global features and local features of the map frame in the visual map, so that the natural features in the image can be fully utilized, which is conducive to improving the accuracy of positioning. The feature matching mode is selected according to the positioning status of the historical image frame adjacent to the current image frame, which is conducive to improving the matching efficiency, thereby improving the efficiency of positioning. Compared with the traditional manually designed feature method, it has strong robustness to lighting changes and weak texture scenes, wide scene adaptability, and can realize the functions of global positioning initialization and repositioning in application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A flow chart of the machine vision positioning method according to an embodiment of the present application.

[0074] Figure 2 A flow chart of the machine vision positioning method according to an embodiment of the present application.

[0075] Figure 3 A schematic diagram of delay calibration and data alignment.

[0076] Figure 4 A schematic diagram of image feature extraction.

[0077] Figure 5 A schematic diagram of the global matching mode.

[0078] Figure 6 A schematic diagram of the prior matching pattern.

[0079] Figure 7 A flow chart for sliding window optimization.

[0080] Figure 8 This is a schematic diagram of a graph structure that graphically describes the data relationship between map feature points, image frame poses, visual observation constraints, and reprojection constraints in this embodiment.

[0081] Figure 9 This is a schematic diagram of a machine vision positioning device according to an embodiment of the present application.

[0082] Figure 10 This is another schematic diagram of the machine vision positioning device according to an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical means and advantages of this application clearer, the application is further described in detail below with reference to the accompanying drawings.

[0084] The embodiment of the present application performs global feature extraction and local feature extraction on the current image frame respectively, and selects a global matching mode or a priori matching mode according to the positioning status of historical image frames adjacent to the current image frame. According to the selected feature matching mode, feature matching is performed on the current image frame with each map frame in the visual map. In this way, both global feature matching and local feature matching are performed, which improves the accuracy of the matched map frame that matches the current image frame, thereby helping to improve the accuracy of positioning. Due to the use of different matching modes, the matching efficiency is greatly improved, which helps to improve the efficiency of positioning.

[0085] See also Figure 1 As shown, Figure 1 A flow chart of the machine vision positioning method according to an embodiment of the present application.

[0086] The method includes:

[0087] Step 101: Obtain image frame data.

[0088] As an example, the image frame data includes: current image frame data, and historical image frame data, for example, a set number of historical image frame data adjacent to the current image frame data.

[0089] Step 102: Perform global feature extraction and local feature extraction on the current image frame to obtain global features and local features respectively.

[0090] in,

[0091] The global feature is used to characterize the characteristic information of the image frame, including: a global feature vector with a first dimension,

[0092] The local feature is used to characterize the image position information of the feature point in the image frame and the feature information of the image space mapped by the feature point, including: the image position information of at least one local feature point and the local feature vector of the local feature point, the local feature vector including: a local feature vector with a second dimension. Since there are usually multiple local feature points, the local feature includes multiple local feature vectors, and each local feature point corresponds to its local feature vector;

[0093] The first dimension and the second dimension may be the same or different.

[0094] As an example, the trained global feature extraction model is used to extract global features of the current image frame to obtain global features.

[0095] Using the trained local feature extraction model, local features are extracted from the current image frame to obtain local features.

[0096] in,

[0097] The global feature extraction model is the first deep learning neural network model,

[0098] The local feature extraction model is the second deep learning neural network model;

[0099] The dimension of the feature vector and the numerical type of the elements depend on the structure of the deep learning neural network model.

[0100] Step 103: Select a feature matching mode based on the positioning status of the historical image frames adjacent to the current image frame. Perform feature matching on the current image frame with each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame.

[0101] Furthermore, it is also possible to obtain pairs of local feature points that match the local feature points in the current image frame with the local feature points in the matching map frame.

[0102] in,

[0103] Each map frame in the visual map is subjected to global feature extraction and local feature extraction respectively, and global features and local features are obtained respectively as map features of the map frame. The extraction method can be the same as that of the image frame;

[0104] The local feature point pairs include: matching local feature points located in the image frame and matching local feature points located in the matching map frame. For the convenience of writing, in the local feature point pairs, the matching local feature points located in the image frame are called image feature points, and the matching local feature points located in the matching map frame are called map feature points.

[0105] The feature matching mode includes: a global matching mode for situations where the positioning state of the historical image frames adjacent to the current image frame fails and / or positioning is initialized (e.g., power on or restarted), and a priori matching mode for situations where the positioning state of the historical image frames adjacent to the current image frame succeeds.

[0106] In the global matching mode, global feature matching is performed first. Since the global feature is a global feature vector with the first dimension, it is equivalent to a coarse matching; then local feature matching is performed. Since the local feature includes the local feature vectors of each local feature point, it is equivalent to a fine matching.

[0107] As an example, in global matching mode, feature matching between the current image frame and each map frame in the visual map includes:

[0108] Match the global features of the current image frame with the global features of each map frame in the visual map,

[0109] Based on the matching results, the candidate map frames with better matching degree are selected.

[0110] For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, the incorrectly matched local feature points are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the candidate map frame.

[0111] Based on the number of matching local feature points of all candidate map frames, the candidate map frame with the largest number of matching local feature points is selected.

[0112] Determine whether the number of matching local feature points of the selected candidate map frame is greater than a set first number threshold. If so, determine that the current image frame is globally matched successfully, and the selected candidate map frame is the matching map frame that matches the current image frame.

[0113] In the prior matching mode, the positioning information of the historical image frames, that is, the posterior pose information of the historical image frames, is used to obtain the prior pose information of the current image frame, and the map frames within the range near the prior pose are selected in the visual map as candidate map frames. This is conducive to quickly locking the candidate map frames and performing local feature matching between the current image frame and the candidate map frames, which is equivalent to performing precise matching.

[0114] As an example, in the prior matching mode, feature matching between the current image frame and each map frame in the visual map includes:

[0115] Based on the posterior pose information of a historical image frame and the change in odometer data between the historical image frame and the current image frame, the prior pose information of the current image frame is calculated, where the prior pose information is the product of the change and the posterior pose.

[0116] Based on the prior pose information, map frames within the vicinity of the prior pose are selected from the visual map as candidate map frames.

[0117] For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, the incorrectly matched local feature points are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the candidate map frame.

[0118] Based on the number of matching local feature points of all candidate map frames, the candidate map frame with the largest number of matching local feature points is selected.

[0119] Determine whether the number of matching local feature points of the selected candidate map frame is greater than a set second number threshold. If so, determine that the current image frame is successfully matched a priori, and the selected candidate map frame is the matching map frame that matches the current image frame.

[0120] or,

[0121] The local feature points of each candidate map frame are merged to generate a local feature point map, and the local feature point map is matched with the current image frame. The local feature points with incorrect matching are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the current image frame.

[0122] Determine whether the number of matching local feature points of the current image frame is greater than the set third number threshold. If so, determine that the current image frame is successfully matched a priori, and use the candidate map frame with the maximum number of inlier matching pairs as the matching map frame.

[0123] Step 104: Determine the positioning result of the current image frame based on the matching map frame.

[0124] Since the matching map frame pose can be obtained when the visual map is constructed, the positioning result of the current image frame can be obtained based on the matching map frame pose.

[0125] Since the current image frame pose determined by matching the map frame is the result of visual matching constraints, when the current image frame has rich features or a small number of matching local feature points, the accuracy of the current image frame pose is poor, so the current image frame pose can be further optimized through step 105.

[0126] Step 105 : Obtain the odometer data corresponding to the image frame data, and optimize the positioning result of the current image frame by fusing the image frame data and the odometer data corresponding to the image frame data.

[0127] As an example, the visual observation constraints between the image frame and its matching map frame, as well as the odometry constraints between the image frames, are used to construct an objective function. The sliding window optimization method is used to solve the objective function and obtain the optimized positioning result.

[0128] in,

[0129] Odometer data includes inertial measurement unit data,

[0130] Visual observation constraints include: the reprojection error between the map feature point and the image feature point in any local feature point pair between the image frame and its matching map frame,

[0131] Odometer constraints include: odometry data error between adjacent image frames,

[0132] The objective function is the minimum value of the sum of the visual observation constraints of all local feature point pairs in the image frame and the sum of all odometry constraints.

[0133] The embodiment of the present application performs both local feature matching and global feature matching on the image frame, which enables fast and accurate positioning in cases of positioning initialization and positioning anomalies, and can realize the functions of global positioning initialization and repositioning in application scenarios; since the image frame includes natural features in the environment, it does not rely on artificial positioning marks and is easier to implement and use. The application of global features and local extraction networks based on deep learning has stronger robustness to lighting changes and weak texture scenes than traditional methods of manually designed features.

[0134] To facilitate understanding of the embodiments of the present application, the following description uses image frames captured by a forward-looking camera and odometer data as examples. It should be understood that the embodiments of the present application are not limited to image frames and odometer data captured by a forward-looking camera, but are also applicable to image frames captured by a downward-looking camera and other inertial measurement unit data.

[0135] What should be understood is that the difference between visual positioning based on forward-looking cameras and downward-looking cameras is reflected in the following: although the image frames captured by forward-looking cameras are affected to a certain extent by lighting, because the camera adopts a forward-looking layout, the effective field of view is large, and more abundant natural features can be observed. The image information is rich, the distinction is good, the scene adaptability is wide, and it is more suitable for weak-texture scenes. This also means that positioning using natural features in image frames does not rely on artificial markers added to the environment, including but not limited to QR codes, ribbons, etc., to facilitate implementation and deployment.

[0136] See also Figure 2 As shown, Figure 2 A flow chart of the machine vision positioning method according to an embodiment of the present application.

[0137] The method includes:

[0138] Step 201: Obtain image frames from a front-view camera and odometer data from an odometer, and align the image frame data with the odometer data so that each image frame data corresponds to odometer data.

[0139] As an example, the odometer is a wheel encoder,

[0140] Generally speaking, the clock sources of the camera and the odometer are inconsistent. In addition, the sampling frequencies are not consistent, and the odometer sampling frequency is usually higher than the camera sampling frequency. Therefore, the data of the two need to be synchronized to facilitate data fusion.

[0141] Data synchronization usually includes: delay calibration and data alignment.

[0142] Delay calibration involves calculating the relative delay between the camera and odometry clocks. This process aligns the camera and odometry sampling domains to the same time domain, aligning the camera and odometry sampling timestamps to the same timestamp system. If camera and odometry timestamp synchronization is implemented at the hardware or software level, delay calibration can be omitted.

[0143] Data alignment aligns the data of the synchronized camera and odometer according to the delay of the delay calibration to obtain the odometer data corresponding to the camera sampling time. In this way, each image frame data can correspond to its odometer data.

[0144] See also Figure 3 As shown, Figure 3A schematic diagram of time delay calibration and data alignment. The camera sampling timestamp information and the odometry sampling timestamp information are unified into the same time domain. In the figure, the hollow circles represent image frame data, and the solid circles represent odometry data. Because the odometry sampling frequency is higher than the camera sampling frequency, the number of odometry sampling points is greater than the number of camera sampling points in the same time.

[0145] When performing data alignment, the odometer data corresponding to the image frame data is matched according to the calibrated delay. As an example, for each image frame sampling time:

[0146] If there is odometer data corresponding to the image frame sampling time, the odometer data is used as the odometer data corresponding to the image frame collected at the image frame sampling time.

[0147] If there is no corresponding odometer data to match the image frame, that is, there is no odometer data corresponding to the image frame sampling time, then interpolation can be performed based on the odometer data collected at the odometer sampling time adjacent to the image frame sampling time, and the odometer data obtained by interpolation can be used as the odometer data corresponding to the image frame collected at the image frame sampling time.

[0148] Similarly, for the current image frame, its corresponding odometer data can also be obtained in a data synchronization manner.

[0149] Step 202: Using the trained global feature extraction network model, perform global feature extraction on the current image frame to reduce the dimension and compress the image frame data to obtain global features. Using the trained local feature extraction network model, perform local feature extraction on the current image frame to obtain local feature point information in the image frame to obtain local features.

[0150] As an example, the global feature extraction network model is a first deep learning neural network model, and the local feature extraction network model is a second deep learning neural network model.

[0151] After the global feature extraction network model performs forward reasoning on the input current image frame, it can output a D g dimensional global descriptor vector The numerical type of the elements of the descriptor vector output by the global feature extraction network model, and the dimension D of the global feature vector descriptor g , both of which can be modified by adjusting the structure of the global feature extraction network.,Wherein, the numerical type of the element can be binary or floating point.

[0152] After the local feature extraction network model performs forward reasoning on the input current image frame, it can output the image position information of N local feature points, such as pixel coordinates p i , and each local feature point pi Corresponding D l dimensional descriptor vector To characterize the image space feature information mapped by the local feature points, where i = 1, 2, 3, ..., N, N is a natural number not less than 1. The number of local feature points N can be set by inputting the configuration parameters of the local feature extraction network model. The dimension D of the local feature point descriptor vector is l The numerical types of the elements are like the global feature extraction network model and can also be modified by adjusting the network structure.

[0153] The global and local features based on deep learning are inherently scale-invariant. To further improve the robustness of global and local features to image scale, an image pyramid approach can be used to extract image feature information at different scales.

[0154] The trained global feature extraction network model and local feature extraction network model can be deployed in mobile robots through transplantation and quantization.

[0155] It should be understood that the training of the global feature extraction network model and the local feature extraction network model can be obtained by training the sample image data collected by the forward-looking camera.

[0156] See also Figure 4 As shown, Figure 4 A schematic diagram of image feature extraction. In the figure, the global feature is D g dimensional vector, a local feature is D for each feature point l dimensional vector, when the number of feature points is N, the local features include N different D l dimensional vector.

[0157] Step 203 : Match the extracted global features and local features with the map features of the map frame in the visual map to establish a one-to-one correspondence between the image features of the image frame and the map features in the visual map.

[0158] For example, when the features of the image frame and the map frame are matched, the local feature points of the image frame and the map feature points in the correct matching pair are imaging observations of the same spatial point in the physical space. In this way, the position and posture of the image frame can be accurately constrained through the camera projection model; conversely, an incorrect matching pair will constitute an incorrect constraint on the position and posture of the image frame, thereby affecting the accuracy of the image frame pose solution and even leading to errors in the image frame pose solution.

[0159] In this embodiment, different feature matching methods are used based on the positioning status of a set number of historical image frames adjacent to the current image frame. Specifically, during positioning initialization and / or after positioning failure occurs during the positioning process, given that there is no accurate estimate of the prior pose of the current image frame, a global matching mode is used for feature matching. When positioning matching of adjacent historical image frames is successful, given that there is a relatively accurate estimate of the pose of the historical image frames, the pose of the historical image frames and the odometer data output by the odometer can be used to obtain a better estimate of the prior pose of the current image frame. In this case, the prior pose is used as an auxiliary prior matching mode.

[0160] As an example, see Figure 5 As shown, Figure 5 This is a diagram of the global matching mode. The global matching mode is as follows:

[0161] First, the global descriptor vector of the current image frame is matched with the global descriptor vector of the map frame in the visual map for global feature matching to perform a rough match. Based on the global descriptor matching results, map frames with better matching scores are selected as candidate map frames, for example, the K candidate map frames with the highest matching scores. The matching scores can be calculated using cosine distance (floating-point descriptor vectors) or Hamming distance (binary descriptor vectors), and K can be selected according to actual conditions.

[0162] Then, all K candidate map frames are traversed, and the local features of the current image frame are matched with the local features of the candidate map frames to perform frame-by-frame point pair matching, that is, the 2D point pair matching of the current image frame and the 3D point pair matching of the candidate map frame is performed. Unmatched external points are eliminated based on the random sampling consensus (RANSAC) method, and the matching pairs of internal points that match each candidate map frame are recorded to obtain local feature point pairs.

[0163] Finally, based on the inlier matching pairs of each candidate map frame, the candidate map frame with the largest number of inlier matching pairs is selected, and it is determined whether the number of inlier matching pairs in the selected candidate map frame is greater than a first number threshold, such as the effective inlier number threshold N inlier_t ,If ,is greater than ,, the global matching and positioning of the current image frame is considered to be successful, and the selected candidate map frame is used as the matching map frame. Otherwise, the global matching and positioning of the current image frame is considered to have failed.

[0164] As an example, see Figure 6 As shown, Figure 6 A schematic diagram of the prior matching mode is shown below.

[0165] Since the premise of selecting this mode is that the positioning matching of the adjacent historical image frames of the current image frame is successful, the current image frame I jThe adjacent historical image frame, for example, the previous image frame I j-1 After reliable pose estimation, we can obtain posterior pose information with high confidence. The corresponding odometry increment between two image frames is ΔT j,j-1 , then the current image frame I is obtained j The prior pose of is the product of the odometer increment and the posterior pose, which can be expressed mathematically as:

[0166]

[0167] Unlike the global matching mode that filters candidate map frames through global descriptors, the prior matching mode uses the current prior pose information Selecting map frames in the vicinity of the prior pose in the visual map as candidate map frames;

[0168] After obtaining each candidate map frame, one method is to use the same method as traversing each candidate map frame in the global matching mode to screen out matching map frames and local feature point pairs, and determine whether the number of inlier matching pairs in the selected candidate map frame is greater than a second number threshold. If so, it is considered that the global matching positioning of the current image frame is successful, and the selected candidate map frame is used as the matching map frame. Otherwise, it is considered that the global matching positioning of the current image frame has failed.

[0169] The second method is that since each candidate map frame is adjacent and continuous in the map space in the prior matching mode, all map feature points associated with each candidate map frame can also be merged to generate a local map of feature points. Then, the local map of feature points is locally matched with the current image frame, and the random sampling consensus (RANSAC) method is used to eliminate mismatched external points to obtain inlier matching pairs of feature points of the current image frame and map feature points as local feature point pairs. It is judged whether the number of inlier matching pairs of the current image frame is greater than a third number threshold. If so, the prior matching positioning of the current image frame is considered to be successful, and the candidate map frame with the largest number of inlier matching pairs is used as the matching map frame. Otherwise, the prior matching positioning of the current image frame is considered to have failed.

[0170] Step 204 : Determine the pose of the current image frame based on the pose information of the matched map frame, and optimize the pose.

[0171] Based on the pose information of the matching map frame, the pose of the current image frame is obtained through visual multi-point PnP solution.

[0172] The pose obtained by the prior matching mode or global matching mode is completely constrained by visual matching and is affected by the current feature richness and the number of feature matches. If there are many mismatches, it is easy to cause positioning failure. To obtain more stable, accurate, and smoother positioning results, the robot pose can be jointly optimized by fusing odometry constraints and visual observation constraints.

[0173] This embodiment adopts a sliding window optimization method, retains a certain amount of key frame information, and uses multiple historical key frames to constrain the positioning results of the current image frame. In this way, even if there are not enough matching feature points, relatively good positioning results can still be obtained, while also effectively balancing computational efficiency and positioning performance.

[0174] See also Figure 7 As shown, Figure 7 A schematic diagram of a sliding window optimization process. Keyframes are selected based on motion distance, but both keyframes and non-keyframes are optimized. The difference is that non-keyframes are deleted after a single optimization, while keyframes remain within the sliding window until they overflow. When an image frame to be optimized enters the sliding window, the earliest keyframe in the window is deleted to enable sliding of the window, as the window contains a fixed number of keyframes (i.e., the window length is fixed). The pose of the image frame to be optimized is then optimized using the image frame information within the window.

[0175] Graph optimization problems that need to be solved for sliding window optimization. Figure 8 As shown, Figure 8 This is a diagram of a graph structure that graphically describes the data relationship between map feature points, image frame poses, visual observation constraints, and reprojection constraints. In the figure, each dot represents a node. The yellow dot represents the matched map feature point, whose pose has been obtained during map construction and remains unchanged during the sliding window optimization. The blue dot represents the pose of the image frame, which is obtained through the visual multi-point PnP solution after the matching is completed. The pose of the image frame is the initial value. Each square represents an edge constraint. The yellow square represents the visual observation constraint of a local feature point pair, specifically, the reprojection constraint r formed between the map feature point and the image feature point in the local feature point pair. v , which can be obtained in the feature matching process; the green square represents the odometry constraint r between the two image frames. o , is the pose increment between two image frames obtained from the odometry data. During the sliding window optimization process, the odometry constraints between image frames and the visual observation constraints of each image frame are integrated to obtain the final pose estimate.

[0176] During the entire sliding window optimization process, because the positions of map feature points are fixed, there are no direct visual constraints between image frames. However, because odometry constraints exist between image frames, the visual constraints of each image frame indirectly and weakly constrain the pose of other image frames. In this embodiment, odometry data is used simply as measurement data, and the odometry state or parameters are not optimized.

[0177] The objective function of the sliding window graph optimization is to find the minimum value of the sum of the visual observation constraints of all local feature point pairs in the image frame and the sum of all odometry constraints. The visual observation constraints and odometry constraints are calculated using the two-norm value, which can be expressed mathematically as follows:

[0178] min{∑‖r o ‖ 2 +∑‖r v ‖ 2}

[0179] in,‖‖ 2 Indicates 2-normal form calculation, r o is the odometry constraint between adjacent keyframes in the sliding window, r v is the reprojection constraint between the map feature point and the image feature point in a local feature point pair within the sliding window, ∑ represents the sum of the constraints on all edges within the sliding window, that is, ∑‖r o ‖ 2 represents the cumulative total odometry constraints, ∑‖r v ‖ 2 Represents the accumulation of all reprojection constraints.

[0180] The optimized pose is obtained by solving the pose of the current image frame that minimizes the objective function. Specifically, if the current image frame is a keyframe, it is added to the sliding window. If the number of keyframes in the sliding window exceeds a set keyframe number threshold, the earliest keyframe in the sliding window is deleted. The objective function is calculated for all keyframes in the current sliding window, and the pose of the current image frame that minimizes the objective function is found.

[0181] The embodiment of the present application performs global feature extraction and local feature extraction on the current image frame from the forward-looking camera, and selects a feature matching mode according to the positioning status of the historical image frame, which is beneficial to improving both the accuracy and efficiency of positioning.

[0182] See also Figure 9 As shown, Figure 9 This is a schematic diagram of a machine vision positioning device according to an embodiment of the present application. The device includes:

[0183] Acquisition module, used to obtain image frame data,

[0184] The feature extraction module is used to extract global features and local features of the current image frame, respectively, to obtain global features and local features.

[0185] The feature matching module is used to select a feature matching mode based on the positioning status of the historical image frames adjacent to the current image frame, and perform feature matching on the current image frame with each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame.

[0186] The positioning module is used to determine the positioning result of the current image frame based on the matching map frame.

[0187] As an example,

[0188] The acquisition module is also used to acquire odometer data corresponding to the image frame data.

[0189] The feature matching module is also used to determine the prior pose information of the current image frame based on the IMU change and the posterior pose in the prior matching mode, so as to quickly select the candidate map frame from the visual map.

[0190] The positioning module is further configured to optimize the positioning result of the current image frame by fusing the image frame data and the odometer data corresponding to the image frame data.

[0191] See also Figure 10 As shown, Figure 10 This is another schematic diagram of a machine vision positioning device according to an embodiment of the present application. The device includes a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the steps of the machine vision positioning method according to an embodiment of the present application.

[0192] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0193] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0194] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the machine vision positioning method described in the embodiment of the present application are implemented.

[0195] As for the apparatus / network-side device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0196] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0197] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A machine vision positioning method, characterized in that: The method includes: Acquire image frame data, the image frame data including: current image frame and historical image frames, Performing global feature extraction and local feature extraction on the current image frame to obtain global features and local features, respectively, wherein the global feature includes: a global feature vector having a first dimension, and the local feature includes: image position information of at least one local feature point and a local feature vector of the local feature point, and the local feature vector includes: a local feature vector having a second dimension, According to the positioning status of the historical image frames adjacent to the current image frame, a feature matching mode is selected. According to the selected feature matching mode, the current image frame is feature matched with each map frame in the visual map to obtain a matching map frame that matches the current image frame. Determine the positioning result of the current image frame based on the matching map frame; in, The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized, and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds. In global matching mode, global feature matching is performed first, followed by local feature matching. In the prior matching mode, the positioning information of the historical image frames is used to obtain the prior pose information of the current image frame, and the map frames within the range near the prior pose are selected in the visual map as candidate map frames for local feature matching.

2. The machine vision positioning method according to claim 1, wherein: The adjacent historical image frames include: a set number of adjacent historical image frames; The extracting global features and local features of the current image frame respectively includes: Using the trained global feature extraction model, global feature extraction is performed on the current image frame to obtain global features. Using the trained local feature extraction model, local features are extracted from the current image frame to obtain local features. in, The global feature extraction model is the first deep learning neural network model, The local feature extraction model is the second deep learning neural network model; The dimension of the feature vector and the numerical type of the elements depend on the structure of the deep learning neural network model.

3. The machine vision positioning method according to claim 2, wherein: In the global matching mode, global feature matching is performed first, and then local feature matching is performed, including: Match the global features of the current image frame with the global features of each map frame in the visual map, Based on the matching results, the candidate map frames with better matching degree are selected. For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, the incorrectly matched local feature points are eliminated, and the correctly matched local feature points are retained to obtain the matched local feature points of the candidate map frame. Based on the number of matching local feature points of all candidate map frames, the candidate map frame with the largest number of matching local feature points is selected. Determine whether the number of matching local feature points of the selected candidate map frame is greater than a set first number threshold. If so, determine that the current image frame is globally matched successfully, and the selected candidate map frame is the matching map frame that matches the current image frame.

4. The machine vision positioning method according to any one of claims 1 to 3, characterized in that: The method of using the positioning information of the historical image frames to obtain the prior pose information of the current image frame includes: Based on the posterior pose information of a historical image frame and the change in odometer data between the historical image frame and the current image frame, the prior pose information of the current image frame is calculated, wherein the prior pose information is the product of the change and the posterior pose. Based on the prior pose information, map frames within a range near the prior pose are selected from the visual map as candidate map frames; The step of selecting a map frame in a vicinity of the prior pose in the visual map as a candidate map frame and performing local feature matching includes: For each candidate map frame, local feature matching is performed between the candidate map frame and the current image frame, local feature points with incorrect matching are eliminated, and correctly matched local feature points are retained to obtain matching local feature points of the candidate map frame. Based on the number of matching local feature points of all candidate map frames, the candidate map frame with the largest number of matching local feature points is selected. It is determined whether the number of matching local feature points of the selected candidate map frame is greater than a set second number threshold. If so, it is determined that the prior matching of the current image frame is successful, and the selected candidate map frame is the matching map frame that matches the current image frame. or, For each candidate map frame, the local feature points of each candidate map frame are merged to generate a local map of feature points. The local map of feature points is matched with the current image frame, the local feature points with incorrect matching are eliminated, and the matching local feature points with correct matching are retained to obtain the matching local feature points of the current image frame. It is determined whether the number of matching local feature points of the current image frame is greater than the set third quantity threshold. If so, it is determined that the prior matching of the current image frame is successful, and the candidate map frame with the largest number of inlier matching pairs is used as the matching map frame.

5. The machine vision positioning method according to claim 4, wherein: The method of removing incorrectly matched local feature points and retaining correctly matched local feature points includes: Using the random sampling consistency algorithm, the matching external points are eliminated to obtain the matching internal points as matching local feature points; The posterior pose information of the historical image frame is the posterior pose information of the previous image frame adjacent to the current image frame; The method further comprises: Get the odometer data corresponding to the image frame data, By fusing the image frame data and the odometer data corresponding to the image frame data, the positioning result of the current image frame is optimized.

6. The machine vision positioning method according to claim 5, wherein: The obtaining of odometer data corresponding to the image frame data includes: Get odometer data, Unify the image sampling time of the acquired image frame data and the odometer sampling time of the acquired odometer data into the same time domain, For each image frame sampling time in the same time domain: If there is odometer data corresponding to the image frame sampling time, the odometer data is used as the odometer data corresponding to the image frame collected at the image frame sampling time. If there is no odometer data corresponding to the image frame sampling time, interpolation is performed based on the odometer data collected at the odometer sampling time adjacent to the image frame sampling time, and the interpolation result is used as the odometer data corresponding to the image frame collected at the image frame sampling time; The optimizing the positioning result of the current image frame by fusing the image frame data and the odometer data corresponding to the image frame data includes: The objective function is constructed using the visual observation constraints between the image frame and its matching map frame, as well as the odometry constraints between the image frames. The sliding window optimization method is used to solve the objective function and obtain the optimized positioning result.

7. The machine vision positioning method according to claim 6, wherein: The objective function constructed by utilizing the visual observation constraint between the image frame and its matching map frame, and the odometry constraint between the image frames, includes: Determine the bi-norm value of the reprojection error between the map feature point and the image feature point in each local feature point pair of the image frame and its matching map frame as the visual observation constraint of the local feature point pair, wherein the local feature point pair is the correctly matched matching local feature point in the image frame and its matching map frame, the matching local feature point in the matching map frame is the map feature point, and the matching local feature point in the image frame is the image feature point. Determine the bi-norm value of the odometry data error between adjacent image frames as an odometry constraint, Accumulate the visual observation constraints of all local feature point pairs, Accumulate all odometry constraints, Sum the accumulated visual observation constraints and the accumulated odometry constraints, Find the minimum value of the summed result as the objective function.

8. The machine vision positioning method according to claim 7, wherein: The image frame data comes from a forward-looking camera installed on the mobile robot body, and the odometer data includes inertial measurement unit data. The sliding window optimization method is used to solve the objective function, including: When the current image frame is a key frame, the current image frame is added to the sliding window. When the number of key frames contained in the sliding window is greater than the set key frame number threshold, the earliest key frame in the sliding window is deleted. Calculate the objective function of all key frames in the current sliding window, Solve the objective function to obtain the current image frame pose, which is the optimized pose.

9. A machine vision positioning device, characterized in that: The device includes: An acquisition module is used to acquire image frame data, wherein the image frame data includes: a current image frame and a historical image frame. The feature extraction module is used to perform global feature extraction and local feature extraction on the current image frame to obtain global features and local features respectively, wherein the global feature includes: a global feature vector with a first dimension, and the local feature includes: image position information of at least one local feature point and a local feature vector of the local feature point, and the local feature vector includes: a local feature vector with a second dimension, The feature matching module is used to select a feature matching mode based on the positioning status of the historical image frames adjacent to the current image frame, and perform feature matching on the current image frame with each map frame in the visual map according to the selected feature matching mode to obtain a matching map frame that matches the current image frame. A positioning module, configured to determine a positioning result of a current image frame based on a matching map frame; in, The feature matching mode includes: a global matching mode for when the positioning state of the historical image frame adjacent to the current image frame fails and / or the positioning is initialized, and a priori matching mode for when the positioning state of the historical image frame adjacent to the current image frame succeeds. In global matching mode, global feature matching is performed first, followed by local feature matching. In the prior matching mode, the positioning information of the historical image frames is used to obtain the prior pose information of the current image frame, and the map frames within the range near the prior pose are selected in the visual map as candidate map frames for local feature matching.

10. A mobile robot, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the machine vision positioning method according to any one of claims 1 to 8.

Images