Dual-target calibration method, apparatus and device, and storage medium

Abstract

The invention discloses a binocular calibration method, device and equipment and a storage medium, and relates to the technical field of cameras, and the method comprises the following steps: obtaining map data of a preset area, a binocular image of a current frame and inertial navigation pose data corresponding to the binocular image; generating a disparity map based on the map data, the binocular image and the inertial navigation pose data; performing feature point matching on the binocular image to obtain a target feature point pair; calculating a predicted parallax value of the target feature point pair based on the parallax map; and adjusting external parameters of the binocular camera based on the predicted parallax value so as to complete binocular calibration. By introducing the map data of the preset area, the external parameters of the binocular camera which is calibrated from a factory are finely adjusted, so that the binocular long-distance ranging precision is improved.

Description

Technical Field

[0001] The present application relates to the field of camera technology, and in particular to a binocular positioning method, device, equipment and storage medium. Background Art

[0002] In order to sense the surrounding environment, flying cars are equipped with multiple sets of binocular devices. These binocular devices must be calibrated at the factory after installation. However, the layout and coverage range of the factory calibration plates are relatively close, resulting in large ranging errors in the binocular calibration results at long distances.

[0003] The current long-distance ranging mainly uses the calibration plate array corner points in the short-range segment for optimization and solution, resulting in poor long-distance ranging accuracy. Summary of the invention

[0004] The main purpose of the present application is to provide a dual-target positioning method, device, equipment and storage medium, aiming to solve the current technical problem of poor long-distance ranging accuracy of dual-target positioning.

[0005] To achieve the above objectives, the present application proposes a dual-target positioning method, which includes:

[0006] Obtaining map data of a preset area, a binocular image of a current frame, and inertial navigation posture data corresponding to the binocular image;

[0007] Generate a disparity map based on the map data, the binocular image and the inertial navigation pose data;

[0008] Performing feature point matching on the binocular image to obtain a target feature point pair;

[0009] Calculating a predicted disparity value of the target feature point pair based on the disparity map;

[0010] The external parameters of the binocular camera are adjusted based on the predicted disparity value to complete binocular positioning.

[0011] In one embodiment, the step of generating a disparity map based on the map data, the binocular image and the inertial navigation pose data comprises:

[0012] Obtaining position information, an intrinsic parameter matrix and an extrinsic parameter matrix of the left camera according to the inertial navigation posture data;

[0013] Converting the map data from the world coordinate system to the flying car coordinate system according to the position information to obtain first map data in the flying car coordinate system;

[0014] Converting the first map data to a left camera coordinate system according to the extrinsic parameter matrix to obtain second map data in the left camera coordinate system;

[0015] The second map data is projected to the left camera image coordinates according to the intrinsic parameter matrix to generate a disparity map.

[0016] In one embodiment, the step of generating a disparity map based on the map data, the binocular image and the inertial navigation pose data comprises:

[0017] Obtaining position information, an intrinsic parameter matrix and an extrinsic parameter matrix of the left camera according to the inertial navigation posture data;

[0018] Converting the map data from the world coordinate system to the flying car coordinate system according to the position information to obtain first map data in the flying car coordinate system;

[0019] Converting the first map data to a left camera coordinate system according to the extrinsic parameter matrix to obtain second map data in the left camera coordinate system;

[0020] The second map data is projected to the left camera image coordinates according to the intrinsic parameter matrix to generate a disparity map.

[0021] In one embodiment, the step of performing feature point matching on the binocular image to obtain a target feature point pair includes:

[0022] Performing feature point detection on the left eye image and the right eye image in the binocular image to generate a feature point descriptor;

[0023] Perform feature point matching between the left image and the right image according to the feature point descriptor to obtain an initial matching point pair;

[0024] The initial matching point pairs are screened to obtain target feature point pairs.

[0025] In one embodiment, the step of screening the initial matching point pairs to obtain target feature point pairs includes:

[0026] Get the rough calibration information of the binocular camera;

[0027] Calculate the essential matrix between the left camera and the right camera in the binocular camera according to the rough calibration information;

[0028] Calculating the epipolar error of the initial matching point pair based on the essential matrix;

[0029] The initial matching point pair whose epipolar error is less than or equal to a preset epipolar error threshold is taken as the target feature point pair.

[0030] In one embodiment, the step of calculating the predicted disparity value of the target feature point pair based on the disparity map comprises:

[0031] Determine, based on the disparity map, a position of a feature point of a left image in the target feature point pair in the disparity map;

[0032] Get the preset search range;

[0033] Taking the feature point position as the center, searching in the disparity map within the preset search range;

[0034] When a valid disparity value is found, a weighted average is performed on the valid disparity value to obtain a predicted disparity value of the left image feature point in the target feature point pair.

[0035] In one embodiment, the step of adjusting the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning includes:

[0036] Dividing the binocular image to obtain a plurality of grid blocks, each grid block corresponding to a preset number of target feature point pairs;

[0037] Get the initial binocular extrinsic parameters obtained by rough calibration;

[0038] Calculate the observed disparity value of the target feature point pair according to the initial binocular extrinsic parameters;

[0039] Selecting a target number of target feature point pairs from the grid blocks, and obtaining target left image feature points according to the target feature point pairs;

[0040] Calculate a residual value according to the predicted disparity value of the target left image feature point and the observed disparity value;

[0041] The initial binocular extrinsic parameters of the binocular camera are adjusted according to the residual value to complete binocular positioning.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a dual-target positioning device, which includes:

[0043] An acquisition module, used to acquire map data of a preset area, a binocular image of a current frame, and inertial navigation posture data corresponding to the binocular image;

[0044] A generating module, configured to generate a disparity map based on the map data, the binocular image and the inertial navigation posture data;

[0045] A matching module, used for matching feature points of the binocular image to obtain a target feature point pair;

[0046] A calculation module, used for calculating the predicted disparity value of the target feature point pair based on the disparity map;

[0047] The adjustment module is used to adjust the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning.

[0048] In addition, to achieve the above objectives, the present application also proposes a dual-target positioning device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the dual-target positioning method described above.

[0049] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the dual-target positioning method described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the dual-target positioning method described above are implemented.

[0051] One or more technical solutions proposed in this application are to obtain map data of a preset area, a binocular image of the current frame, and the inertial navigation posture data corresponding to the binocular image; generate a disparity map based on the map data, the binocular image, and the inertial navigation posture data; perform feature point matching on the binocular image to obtain a target feature point pair; calculate the predicted disparity value of the target feature point pair based on the disparity map; and adjust the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning. By introducing the map data of the preset area, the external parameters of the factory-calibrated binocular camera can be fine-tuned to improve the accuracy of binocular long-distance ranging. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the dual-target positioning method of the present application;

[0055] Figure 2 A schematic diagram of a flow chart provided for the second embodiment of the dual-target positioning method of the present application;

[0056] Figure 3 A schematic diagram of a process flow provided for the third embodiment of the dual-target positioning method of the present application;

[0057] Figure 4 A schematic diagram of a flow chart provided for the fourth embodiment of the dual-target positioning method of the present application;

[0058] Figure 5 A schematic diagram of a brief flow chart of a dual-target positioning method provided in one embodiment of the present application;

[0059] Figure 6 This is a schematic diagram of the module structure of the dual-target positioning device according to an embodiment of the present application;

[0060] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the dual-target positioning method in the embodiment of the present application.

[0061] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0063] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0064] The main solution of the embodiment of the present application is: obtaining map data of a preset area, a binocular image of a current frame and inertial navigation posture data corresponding to the binocular image; generating a disparity map based on the map data, the binocular image and the inertial navigation posture data; performing feature point matching on the binocular image to obtain a target feature point pair; calculating a predicted disparity value of the target feature point pair based on the disparity map; and adjusting the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning.

[0065] The existing technology is to arrange various calibration plate arrays with different postures within a certain distance range (generally about 1-4m) in the calibration factory workshop, and then optimize the binocular calibration results through Zhang Zhengyou calibration method. Since the detection distance of the binocular equipment needs to cover up to 200m, the coverage range of the calibration plate array in the general calibration scene cannot be too different from the coverage range of the use scene, otherwise it is easy to cause large ranging errors at long distances. However, it is difficult to set up dense artificial targets at a position greater than 10 meters in the calibration factory workshop. Therefore, the existing technology only uses the corner points of the calibration plate array in the short-range segment for optimization and solution, resulting in poor long-distance ranging accuracy.

[0066] This application provides a solution that does not require the additional setting of dense artificial targets at a distance greater than 10 meters. It fine-tunes the factory calibration by introducing high-precision maps, and uses parallax as the residual to optimize the calibration results, thereby improving the accuracy of binocular long-distance ranging.

[0067] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a dual-target positioning device, etc. The following takes the dual-target positioning device as an example to illustrate this embodiment and the following embodiments.

[0068] Based on this, the present application embodiment provides a dual-target positioning method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the dual-target positioning method of the present application.

[0069] In this embodiment, the dual-target positioning method includes steps S10 to S40:

[0070] Step S10: obtaining map data of a preset area, a binocular image of a current frame, and inertial navigation posture data corresponding to the binocular image.

[0071] It should be noted that the preset area is the area where the flying car is to fly. The flying position of the flying car can be planned in advance, so that the preset area can be determined according to the flying position. The map data of the preset area is high-precision map data, which may include geographical information, environmental characteristics and other information. After the preset area is determined, the map data of the preset area can be preloaded. The map data can also be obtained by detecting the surrounding environment information through sensors such as laser radar.

[0072] In a specific implementation, continuous image frame data can be collected during the flight of the flying car, so as to obtain the binocular image of the current frame and the inertial navigation posture data corresponding to the binocular image. The binocular image is the left image and the right image. The binocular image can be obtained by taking pictures with the left and right cameras installed on the flying car.

[0073] The inertial navigation posture data includes information such as the position, speed, attitude, acceleration, and angular velocity of the flying car, as well as the intrinsic and extrinsic matrix of the left camera and the intrinsic and extrinsic matrix of the right camera.

[0074] Step S20: generating a disparity map based on the map data, the binocular image and the inertial navigation posture data.

[0075] It should be noted that the map data can be projected onto the left camera image in the binocular image based on the inertial navigation posture data, so as to generate a disparity map representing the depth information based on the projection result. The disparity map is the disparity map of the high-precision map. The disparity map reflects the distance of the object from the camera.

[0076] Step S30: performing feature point matching on the binocular image to obtain a target feature point pair.

[0077] It should be understood that feature extraction can be performed on the left and right images to identify significant visual feature points, and feature point matching can be performed to obtain target feature point pairs that meet the requirements. The target feature point pairs are the corresponding multiple groups of feature point pairs in the left and right images.

[0078] Step S40: calculating the predicted disparity value of the target feature point pair based on the disparity map.

[0079] In a specific implementation, the predicted disparity value of the target feature point pair can be calculated based on the disparity map, that is, the predicted disparity value of the target feature point pair on the high-precision map disparity map.

[0080] In this embodiment, the predicted disparity value of the target feature point pair is mainly the predicted disparity value of the left image feature point in the target feature point pair.

[0081] Since the disparity map of the high-precision map does not necessarily guarantee that there will be disparity values ​​at the corresponding feature point positions, it is necessary to screen the feature points in the left image to obtain the predicted disparity values ​​of the target feature points for the feature points in the left image.

[0082] In a feasible implementation, step S40 may include steps S41 to S44:

[0083] Step S41: determining the position of the feature point of the left image feature point in the target feature point pair in the disparity map based on the disparity map.

[0084] It can be understood that, since the disparity map includes the positions corresponding to the feature points in the left image and the right image, the positions of the feature points of the left image in the target feature point pair in the disparity map can be obtained.

[0085] Step S42: Obtain a preset search range.

[0086] Since there may not be a predicted disparity value at every feature point, a search may be performed around the corresponding disparity map centered on the feature point. The preset search range may be set to ±1px and may be adjusted according to the size of the disparity map or the distribution of feature points.

[0087] Step S43: searching the disparity map in the preset search range with the feature point position as the center.

[0088] In a specific implementation, each feature point position is taken as the center, and a search is performed on the disparity map within a preset search range to determine whether the disparity value corresponding to the feature point can be searched. If it is found, the disparity value is used as a valid disparity value. If it is not found, it means that there is no valid disparity value around the feature point, and the feature point is removed.

[0089] Step S44: when a valid disparity value is found, weighted averaging is performed on the valid disparity value to obtain a predicted disparity value of the left image feature point in the target feature point pair.

[0090] In a specific implementation, if a valid disparity value is found, all valid disparity values ​​found within a preset search range may be accumulated and averaged to obtain a predicted disparity value of the left image feature point in the target feature point pair.

[0091] The process of calculating the predicted disparity value is as follows:

[0092]

[0093] In the above formula 1, disp is the predicted disparity value, i and j, which represent the row and column indices respectively. Here, i ranges from y-1 to y+1, and j ranges from x-1 to x+1. This means that we are looking at a 3x3 area, which includes our center point P(x,y), and Dmap(i,j) refers to the disparity value at position (i,j), that is, the effective disparity value. By accumulating and averaging the effective disparity values, the average disparity value is obtained, which is used as the predicted disparity value of the feature point on the left.

[0094] Step S50: adjusting the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning.

[0095] After obtaining the predicted disparity value, the external parameters of the binocular camera can be adjusted according to the predicted disparity value. For example, the observed disparity value of the feature point pair is calculated according to the coarsely calibrated external parameters, and the residual value is calculated according to the observed disparity value and the predicted disparity value. The external parameters of the binocular camera are adjusted according to the residual value so that the predicted disparity value is equal to the observed disparity value, and the binocular positioning is completed, thereby improving the long-distance ranging accuracy.

[0096] This embodiment provides a binocular positioning method, which obtains map data of a preset area, a binocular image of a current frame, and inertial navigation posture data corresponding to the binocular image; generates a disparity map based on the map data, the binocular image, and the inertial navigation posture data; performs feature point matching on the binocular image to obtain a target feature point pair; calculates a predicted disparity value of the target feature point pair based on the disparity map; and adjusts the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning. By introducing the map data of the preset area, the external parameters of the factory-calibrated binocular camera are fine-tuned, thereby improving the accuracy of binocular long-distance ranging.

[0097] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2, step 20 includes steps S201 to S204:

[0098] Step S201: obtaining position information, an intrinsic parameter matrix and an extrinsic parameter matrix of a left camera according to the inertial navigation posture data.

[0099] It should be noted that the inertial navigation pose data includes position information, the intrinsic parameter matrix of the left and right cameras, and the extrinsic parameter matrix. Therefore, the position information, the intrinsic parameter matrix of the left camera, and the extrinsic parameter matrix of the left camera can be obtained according to the inertial navigation pose data. The intrinsic parameter matrix includes parameters such as focal length and principal point coordinates.

[0100] Step S202: converting the map data from the world coordinate system to the flying car coordinate system according to the position information to obtain first map data in the flying car coordinate system.

[0101] In a specific implementation, the map data can be converted from the world coordinate system to the flying car coordinate system according to the position and direction of the left camera, so as to obtain the map data relative to the current position and posture of the flying car, that is, the first map data.

[0102] The geographic coordinates (usually longitude, latitude and altitude) on the high-precision map first need to be converted into a local Cartesian coordinate system, the origin of which is usually set at the current position of the flying car. The purpose of this is to convert the geographic coordinates into relative coordinates related to the flying car for subsequent processing.

[0103] Step S203: converting the first map data into a left camera coordinate system according to the extrinsic parameter matrix to obtain second map data in the left camera coordinate system.

[0104] It can be understood that after obtaining the first map data, the external parameter matrix from the flying car to the left camera can be used to convert the high-precision map data in the flying car coordinate system to the left camera coordinate system, thereby obtaining the map data of the first map data in the left camera coordinate system, that is, the second map data.

[0105] Step S204: projecting the second map data to the left camera image coordinates according to the intrinsic parameter matrix to generate a disparity map.

[0106] It should be noted that the map data points in the left coordinate system can be projected onto the image plane of the left camera according to the intrinsic parameter matrix of the left camera, and then the disparity value of the pixel point of each map data point on the image plane can be generated, and the disparity map can be generated according to the disparity value.

[0107] In a feasible implementation, step S204 may include steps A11 to A14:

[0108] Step A11: Project the second map data to the left camera image coordinates according to the intrinsic parameter matrix to obtain map pixel point data.

[0109] In a specific implementation, the second map data in the left camera coordinate system can be projected onto the image plane of the left camera according to the intrinsic parameter matrix, thereby obtaining the pixel coordinates corresponding to each map data point on the left camera image plane, that is, the map pixel point data.

[0110] Step A12: Obtain the focal length of the left camera, the binocular baseline distance, and the depth value of the map pixel data.

[0111] In the specific implementation, in the left camera image coordinates, the depth value of each map pixel data can be directly obtained, and at the same time, the left camera focal length f and the binocular baseline distance b can also be obtained. The depth value is the straight-line distance from the origin of the camera coordinate system to the map data point, the baseline distance b refers to the horizontal distance between the two cameras in the binocular camera system, and the camera focal length f can be obtained from the intrinsic parameter matrix of the left camera.

[0112] Step A13: Calculate the pixel disparity value according to the depth value, the left camera focal length and the binocular baseline distance.

[0113] In a specific implementation, the pixel disparity value can be calculated according to the depth value, the focal length f of the left camera and the binocular baseline distance b. The calculation process is as follows:

[0114]

[0115] In the above formula 2, d is the pixel disparity value, f is the focal length of the left camera, b is the binocular baseline distance, P z is the depth value of each pixel.

[0116] Step A14: Integrate the disparity values ​​of the pixel points to obtain a disparity map of the map data.

[0117] In a specific implementation, the disparity values ​​of all pixel points can be integrated to generate a disparity map of the final map data. In the above manner, all points of the high-precision map are converted to the left camera image to generate a disparity map based on the high-precision map.

[0118] This embodiment obtains position information, the intrinsic parameter matrix and the extrinsic parameter matrix of the left camera according to the inertial navigation posture data; converts the map data from the world coordinate system to the flying car coordinate system according to the position information to obtain the first map data in the flying car coordinate system; converts the first map data to the left camera coordinate system according to the extrinsic parameter matrix to obtain the second map data in the left camera coordinate system; projects the second map data to the left camera image coordinates according to the intrinsic parameter matrix to generate a disparity map. By combining the inertial navigation posture data with the map data, the positioning accuracy of the flying car in a complex environment can be improved, and reliable positioning information can be provided. The intrinsic parameter matrix is ​​used to project the second map data to the left camera image coordinates to generate a disparity map. The disparity map can provide depth information, help the flying car better understand the three-dimensional space structure, and plays an important role in tasks such as obstacle avoidance and target tracking, thereby improving the effect of binocular positioning.

[0119] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step 30 includes steps S301 to S303:

[0120] Step S301: performing feature point detection on the left eye image and the right eye image in the binocular image to generate feature point descriptors.

[0121] It should be noted that, feature point detection for the left eye image and the right eye image in the binocular image can be performed by using corner detection, edge detection or other existing image feature detection methods. For example, a feature description algorithm (OrientedFAST and Rotated BRIEF, ORB) can be used for feature point detection. The ORB algorithm combines the FAST corner detection algorithm and the BRIEF descriptor algorithm to achieve efficient and rotation-invariant feature extraction and matching. The ORB algorithm is used to detect feature points in the left eye image and the right eye image respectively, and generate corresponding BRIEF descriptors.

[0122] In the specific implementation, the threshold and maximum number of feature points for ORB feature point detection can be determined, and the FAST corner detector can be used to find candidate feature points. By calculating the direction information of each candidate feature point, rotation invariance can be achieved, and the target feature point, that is, the most significant feature point, is selected based on the calculated direction information, and a BRIEF descriptor is generated for each target feature point.

[0123] Step S302: performing feature point matching between the left-eye image and the right-eye image according to the feature point descriptor to obtain an initial matching point pair.

[0124] It can be understood that feature matching can be performed between the left-eye image and the right-eye image based on the generated feature point descriptor. Specifically, the matching parameters can be set, so that matching can be performed according to the fast nearest neighbor search (FLANN) and the set matching parameters, and a set of matched feature point pairs, i.e., the initial matching point pairs, can be output.

[0125] Step S303: Screening the initial matching point pairs to obtain target feature point pairs.

[0126] In a specific implementation, in order to reduce the impact of mismatching, the initial matching point pairs may be screened to remove feature point pairs that do not meet the conditions, thereby obtaining the screened feature point pairs, ie, the target feature point pairs.

[0127] In a feasible implementation, step S303 may include steps B11 to B14:

[0128] Step B11: Obtain the rough calibration information of the binocular camera.

[0129] It should be noted that the coarse calibration information refers to the information determined by the binocular camera during coarse calibration, and the coarse calibration refers to the result of calibration of the calibration plate array.

[0130] The rough calibration information may include the intrinsic parameter matrix, relative rotation matrix, and translation vector of the binocular camera.

[0131] Step B12: Calculate the essential matrix between the left camera and the right camera in the binocular camera according to the rough calibration information.

[0132] In a specific implementation, the intrinsic matrix E between the two cameras in the binocular camera can be obtained by calculation based on the intrinsic parameter matrices K_left and K_right, the relative rotation matrix R, and the translation vector t in the coarse calibration information.

[0133] Step B13: Calculate the epipolar error of the initial matching point pair based on the essential matrix.

[0134] It can be understood that the epipolar error of the initial matching point pair can be calculated according to the essential matrix E, as shown in the following formula 3:

[0135]

[0136] In the above formula 3, the essential matrix E is a 3×3 matrix, p r and p l are the homogeneous coordinates of the corresponding points in the right image and the left image respectively. The epipolar error Err of each pair of feature points can be calculated by the above formula 3.

[0137] Step B14: taking the initial matching point pairs whose epipolar errors are less than or equal to a preset epipolar error threshold as target feature point pairs.

[0138] In a specific implementation, a preset epipolar error threshold thr can be set, such as 0.2, 0.3, etc., and the epipolar error is compared with the preset epipolar error threshold. If p r and p l is the true corresponding point, then the epipolar error should be close to zero. Therefore, if the epipolar error is greater than the preset epipolar error threshold, the initial matching point pair corresponding to this epipolar error is eliminated, and the initial matching point pair with the epipolar error less than or equal to the preset epipolar error threshold is retained as the target feature point pair.

[0139] This embodiment generates a feature point descriptor by performing feature point detection on the left eye image and the right eye image in the binocular image; performs feature point matching between the left eye image and the right eye image according to the feature point descriptor to obtain an initial matching point pair; and screens the initial matching point pair to obtain a target feature point pair. By performing feature point detection on the binocular image, the key area in the image can be accurately located, and the matched feature points can be screened to reduce the impact of mismatching, so that the disparity value can be calculated based on the target feature point pair in the subsequent process, thereby improving the optimization effect of binocular positioning.

[0140] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. Figure 4 , step 50 includes steps S501 to S506:

[0141] Step S501: Divide the binocular image to obtain a plurality of grid blocks, each grid block corresponding to a preset number of target feature point pairs.

[0142] In order to make the number of feature points uniform in the entire image, the binocular image can be evenly divided. Specifically, the grid block (M*N) can be evenly divided to obtain multiple grid blocks. The preset number N can be set to a value greater than X. For example, if X is set to 20, the preset number is greater than 20. When dividing, if the number of target feature point pairs in the divided grid block is less than or equal to 20, return to the step of obtaining the binocular image and the corresponding inertial navigation pose data until the number of divided target feature point pairs is greater than the preset number.

[0143] Step S502: Obtain initial binocular extrinsic parameters obtained through rough calibration.

[0144] In a specific implementation, the initial binocular extrinsic parameters in the coarse calibration information may be obtained, that is, the extrinsic parameters of the left and right cameras.

[0145] Step S503: Calculate the observed disparity value of the target feature point pair according to the initial binocular extrinsic parameters.

[0146] In the specific implementation, the observed disparity value of the target feature point pair can be calculated according to the initial binocular external parameters, that is, the actual disparity value. Specifically, the intrinsic parameters and distortion coefficients of the left and right cameras can be used to transform them into the camera coordinate system, which are the left camera coordinate system P l and the right camera coordinate system P r , and then transform the rotation matrix R from the left camera coordinate system to the right camera coordinate system according to the rough calibration external parameters. The formula is as follows to calculate the transformation of the left camera coordinate system to the right camera coordinate system P l ', the calculation process is as follows:

[0147] P l '=R*P l (Formula 4)

[0148] The observed parallax value is calculated by converting the abscissa value of the point in the right camera coordinate system into the abscissa value of the point in the right camera coordinate system through the above formula 4, as shown in the following formula 5:

[0149] Dobserved=P' lx -P rx (Formula 5)

[0150] In the above formula 5, the observed disparity value of the target feature point pair is Dobserved.

[0151] Step S504: selecting a target number of target feature point pairs from the grid blocks, and obtaining target left-image feature points according to the target feature point pairs.

[0152] In a specific implementation, target feature point pairs may be randomly selected from each grid block, the target number being X, and the target number of target left-image feature points may be obtained based on the selected target number of target feature point pairs.

[0153] Step S505: Calculate a residual value according to the predicted disparity value of the target left image feature point and the observed disparity value.

[0154] In a specific implementation, the predicted disparity value of the feature point of the target left image on the disparity map can be obtained, so that the residual value can be calculated according to the predicted disparity value and the observed disparity value, which is calculated as follows:

[0155]

[0156] In the above formula (6), N is the number of feature points after homogenization, Dobserved is the observed disparity value, and Dperdicted is the predicted disparity value.

[0157] Step S506: adjusting the initial binocular extrinsic parameters of the binocular camera according to the residual value to complete binocular positioning.

[0158] In a specific implementation, a residual threshold can be set to compare the residual value with the residual threshold. When the residual value is greater than the residual threshold, the initial binocular extrinsic parameters of the binocular camera are adjusted until the residual value is less than or equal to the residual threshold, thereby optimizing the calibration of the binocular camera.

[0159] This embodiment divides the binocular image to obtain a plurality of grid blocks, each of which corresponds to a preset number of target feature point pairs; obtains the initial binocular external parameters obtained by rough calibration; calculates the observed disparity value of the target feature point pairs based on the initial binocular external parameters; selects a target number of target feature point pairs from the grid blocks, and obtains target left image feature points based on the target feature point pairs; calculates residual values ​​based on the predicted disparity values ​​of the target left image feature points and the observed disparity values; and adjusts the initial binocular external parameters of the binocular camera based on the residual values ​​to complete binocular positioning. By optimizing the external parameters of the binocular system to minimize the difference between the two, the optimized external parameters of the binocular system are output for subsequent vehicle positioning and navigation to improve the binocular positioning effect.

[0160] For example, to help understand the implementation process of the dual-target positioning method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 5 , Figure 5 A brief flow chart of a binocular positioning method is provided, specifically: pre-loading a high-precision map of the area, and obtaining the left and right binocular images and corresponding inertial navigation pose data of the current frame; projecting the high-precision map data onto the left camera image to generate a disparity map; performing feature point detection and matching on the left and right binocular images, and eliminating erroneous feature point pairs according to the epipolar error threshold of coarse calibration; obtaining predicted disparity values ​​on the projected map disparity map according to the feature points of the left image; evenly dividing the image into M*N grid blocks, and storing the feature point pairs in the corresponding blocks; judging whether the minimum number of feature point pairs in each grid block is greater than X, if not, returning to the step of obtaining the left and right binocular images and corresponding inertial navigation pose data of the current frame, if yes, randomly extracting feature point pairs for each grid block, and keeping the number of feature point pairs for each grid block as X; then calculating the observed disparity values ​​of the feature point pairs with the initial binocular extrinsic parameters in the coarse calibration, and taking the map disparity values ​​as the predicted disparity values, thereby optimizing and fine-tuning the binocular extrinsic parameters.

[0161] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the dual-target positioning method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0162] This application also provides a dual-target positioning device, please refer to Figure 6, the dual-target positioning device comprises:

[0163] An acquisition module 10 is used to acquire map data of a preset area, a binocular image of a current frame, and inertial navigation posture data corresponding to the binocular image;

[0164] A generating module 20, configured to generate a disparity map based on the map data, the binocular image and the inertial navigation posture data;

[0165] A matching module 30 is used to perform feature point matching on the binocular image to obtain a target feature point pair;

[0166] A calculation module 40, configured to calculate a predicted disparity value of the target feature point pair based on the disparity map;

[0167] The adjustment module 50 is used to adjust the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning.

[0168] The binocular positioning device provided by the present application adopts the binocular positioning method in the above embodiment, which can solve the technical problem of poor accuracy of long-distance ranging of binocular positioning. Compared with the prior art, the beneficial effects of the binocular positioning device provided by the present application are the same as the beneficial effects of the binocular positioning method provided by the above embodiment, and other technical features in the binocular positioning device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0169] In one embodiment, the generation module 20 is further used to obtain position information, an intrinsic parameter matrix and an extrinsic parameter matrix of the left camera according to the inertial navigation posture data; convert the map data from the world coordinate system to the flying car coordinate system according to the position information to obtain first map data in the flying car coordinate system; convert the first map data to the left camera coordinate system according to the extrinsic parameter matrix to obtain second map data in the left camera coordinate system; project the second map data to the left camera image coordinates according to the intrinsic parameter matrix to generate a disparity map.

[0170] In one embodiment, the generation module 20 is also used to project the second map data to the left camera image coordinates according to the intrinsic parameter matrix to obtain map pixel data; obtain the left camera focal length, the binocular baseline distance and the depth value of the map pixel data; calculate the pixel disparity value according to the depth value, the left camera focal length and the binocular baseline distance; integrate the pixel disparity values ​​to obtain a disparity map of the map data.

[0171] In one embodiment, the matching module 30 is further used to perform feature point detection on the left eye image and the right eye image in the binocular image to generate a feature point descriptor; perform feature point matching between the left eye image and the right eye image according to the feature point descriptor to obtain an initial matching point pair; and screen the initial matching point pair to obtain a target feature point pair.

[0172] In one embodiment, the matching module 30 is also used to obtain rough calibration information of the binocular camera; calculate the intrinsic matrix between the left camera and the right camera in the binocular camera according to the rough calibration information; calculate the epipolar error of the initial matching point pair based on the intrinsic matrix; and use the initial matching point pair whose epipolar error is less than or equal to a preset epipolar error threshold as the target feature point pair.

[0173] In one embodiment, the calculation module 40 is also used to determine the feature point position of the left image feature point in the target feature point pair in the disparity map based on the disparity map; obtain a preset search range; search the disparity map within the preset search range with the feature point position as the center; when a valid disparity value is searched, perform weighted averaging on the valid disparity value to obtain a predicted disparity value of the left image feature point in the target feature point pair.

[0174] In one embodiment, the adjustment module 50 is further used to divide the binocular image to obtain a plurality of grid blocks, each grid block corresponding to a preset number of target feature point pairs; obtain initial binocular extrinsic parameters obtained by rough calibration; calculate observed disparity values ​​of the target feature point pairs based on the initial binocular extrinsic parameters; select a target number of target feature point pairs from the grid blocks, and obtain target left-image feature points based on the target feature point pairs; calculate residual values ​​based on the predicted disparity values ​​of the target left-image feature points and the observed disparity values; and adjust the initial binocular extrinsic parameters of the binocular camera based on the residual values ​​to complete binocular calibration.

[0175] The present application provides a dual-target positioning device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the dual-target positioning method in the above-mentioned embodiment 1.

[0176] Reference below Figure 7, which shows a schematic diagram of the structure of a dual-target positioning device suitable for implementing the embodiment of the present application. The dual-target positioning device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The binocular positioning device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0177] like Figure 7 As shown, the dual-target positioning device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the dual-target positioning device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the dual-target positioning device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a dual-target positioning device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0178] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0179] The binocular positioning device provided by the present application adopts the binocular positioning method in the above embodiment, which can solve the technical problem of poor accuracy of long-distance ranging of binocular positioning. Compared with the prior art, the beneficial effects of the binocular positioning device provided by the present application are the same as the beneficial effects of the binocular positioning method provided by the above embodiment, and other technical features in the binocular positioning device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0180] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0181] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0182] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the dual-target positioning method in the above-mentioned embodiment.

[0183] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0184] The computer-readable storage medium may be included in the dual-target positioning device; or may exist independently without being assembled into the dual-target positioning device.

[0185] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the binocular positioning device, the binocular positioning device: obtains map data of a preset area, a binocular image of a current frame, and inertial navigation posture data corresponding to the binocular image; generates a disparity map based on the map data, the binocular image, and the inertial navigation posture data; performs feature point matching on the binocular image to obtain a target feature point pair; calculates a predicted disparity value of the target feature point pair based on the disparity map; and adjusts the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning.

[0186] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0188] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0189] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned dual-target positioning method, and can solve the technical problem of poor accuracy of long-distance ranging in dual-target positioning. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the dual-target positioning method provided by the above-mentioned embodiment, and will not be described in detail here.

[0190] The present application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the dual-target positioning method as described above are implemented.

[0191] The computer program product provided by this application can solve the current technical problem of poor accuracy of long-distance ranging of dual-target positioning. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the dual-target positioning method provided by the above embodiment, which will not be repeated here.

[0192] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A binocular positioning method, characterized in that: The dual-target positioning method comprises: Obtaining map data of a preset area, a binocular image of a current frame, and inertial navigation posture data corresponding to the binocular image; Generate a disparity map based on the map data, the binocular image and the inertial navigation pose data; Performing feature point matching on the binocular image to obtain a target feature point pair; Calculating a predicted disparity value of the target feature point pair based on the disparity map; The external parameters of the binocular camera are adjusted based on the predicted disparity value to complete binocular positioning.

2. The method according to claim 1, characterized in that The step of generating a disparity map based on the map data, the binocular image and the inertial navigation pose data comprises: Obtaining position information, an intrinsic parameter matrix and an extrinsic parameter matrix of the left camera according to the inertial navigation posture data; Converting the map data from the world coordinate system to the flying car coordinate system according to the position information to obtain first map data in the flying car coordinate system; Converting the first map data to a left camera coordinate system according to the extrinsic parameter matrix to obtain second map data in the left camera coordinate system; The second map data is projected to the left camera image coordinates according to the intrinsic parameter matrix to generate a disparity map.

3. The method according to claim 2, characterized in that The step of projecting the second map data to the left camera image coordinates according to the intrinsic parameter matrix to generate a disparity map comprises: Projecting the second map data to the left camera image coordinates according to the intrinsic parameter matrix to obtain map pixel point data; Obtain the focal length of the left camera, the binocular baseline distance, and the depth value of the map pixel data; Calculate the pixel disparity value according to the depth value, the focal length of the left camera and the binocular baseline distance; The disparity values ​​of the pixel points are integrated to obtain a disparity map of the map data.

4. The method according to claim 1, characterized in that The step of performing feature point matching on the binocular image to obtain a target feature point pair comprises: Performing feature point detection on the left eye image and the right eye image in the binocular image to generate a feature point descriptor; Perform feature point matching between the left image and the right image according to the feature point descriptor to obtain an initial matching point pair; The initial matching point pairs are screened to obtain target feature point pairs.

5. The method according to claim 4, characterized in that The step of screening the initial matching point pairs to obtain target feature point pairs comprises: Get the rough calibration information of the binocular camera; Calculate the essential matrix between the left camera and the right camera in the binocular camera according to the rough calibration information; Calculating the epipolar error of the initial matching point pair based on the essential matrix; The initial matching point pair whose epipolar error is less than or equal to a preset epipolar error threshold is taken as the target feature point pair.

6. The method according to claim 1, characterized in that The step of calculating the predicted disparity value of the target feature point pair based on the disparity map comprises: Determine, based on the disparity map, a position of a feature point of a left image in the target feature point pair in the disparity map; Get the preset search range; Taking the feature point position as the center, searching in the disparity map within the preset search range; When a valid disparity value is found, a weighted average is performed on the valid disparity value to obtain a predicted disparity value of the left image feature point in the target feature point pair.

7. The method according to any one of claims 1 to 6, characterized in that The step of adjusting the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning includes: Dividing the binocular image to obtain a plurality of grid blocks, each grid block corresponding to a preset number of target feature point pairs; Get the initial binocular extrinsic parameters obtained by rough calibration; Calculate the observed disparity value of the target feature point pair according to the initial binocular extrinsic parameters; Selecting a target number of target feature point pairs from the grid blocks, and obtaining target left image feature points according to the target feature point pairs; Calculate a residual value according to the predicted disparity value of the target left image feature point and the observed disparity value; The initial binocular extrinsic parameters of the binocular camera are adjusted according to the residual value to complete binocular positioning.

8. A binocular positioning device, characterized in that: The device comprises: An acquisition module, used to acquire map data of a preset area, a binocular image of a current frame, and inertial navigation posture data corresponding to the binocular image; A generating module, configured to generate a disparity map based on the map data, the binocular image and the inertial navigation posture data; A matching module, used for matching feature points of the binocular image to obtain a target feature point pair; A calculation module, used for calculating the predicted disparity value of the target feature point pair based on the disparity map; The adjustment module is used to adjust the external parameters of the binocular camera based on the predicted disparity value to complete binocular positioning.

9. A binocular targeting device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the dual-target positioning method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the dual-target positioning method according to any one of claims 1 to 7 are implemented.

Images