Binocular matching method for SLAM positioning and virtual reality equipment

Abstract

The invention relates to the technical field of VR, provides a binocular matching method for SLAM positioning and virtual reality equipment, and is used for improving the precision and real-time performance of SLAM positioning. According to the method, for each first feature point extracted from a first environment image collected by a first camera, based on a gray scale invariance hypothesis, depth value coarse search is carried out in a depth range of the first feature point at a preset step length; obtaining an initial depth enabling gray difference between a point in a preset field of the first feature point and a homonymy point in a second environment image collected by the second camera to be minimum, performing linear optimization on the initial depth by adopting a least square method to obtain an accurate target depth, and combining pre-calibrated external parameters between binocular cameras to obtain a target depth. According to the method, feature point matching pairs of SLAM positioning are obtained, so that selection and scale differences caused by different camera orientations are overcome, the feature matching precision is improved, image correction is not needed, and the real-time performance of SLAM positioning is ensured.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology, and provides a binocular matching method and virtual reality equipment for SLAM positioning. Background Art

[0002] During the use of VR devices, the simultaneous localization and mapping (SLAM) technology is used to obtain the 6-degree-of-freedom (Dof) posture. During the positioning process, in order to restore the scale information of SLAM, the VR device is equipped with a camera that provides observation data for SLAM in addition to an inertial measurement unit (IMU) that provides motion measurement data for SLAM. In order to ensure that more environmental features for calculating the 6DoF posture are obtained during the operation of SLAM, the number of cameras equipped is usually multiple, such as binocular cameras or quad cameras (i.e., two pairs of binocular cameras).

[0003] In order to avoid the field of view of multiple cameras being completely blocked, the cameras are not kept parallel, but have a certain angle to maintain a large difference in orientation, such as Figure 1 As shown in the figure, although such a design maximizes the field of view (FOV) range of the camera from the hardware structure and solves the problem of camera occlusion, it also increases the difficulty of feature matching between cameras. This is because the difference in orientation between cameras makes the rotation and scale of the collected images inconsistent. The same feature point will show different rotations and scales in the images collected by different cameras at the same time, which affects the feature matching results and further affects the positioning accuracy of SLAM. Summary of the invention

[0004] The embodiments of the present application provide a binocular matching method and a virtual reality device for SLAM positioning, which improve the positioning accuracy of SLAM by improving the accuracy of binocular matching.

[0005] On the one hand, an embodiment of the present application provides a binocular matching method for SLAM positioning, which is applied to a virtual reality device, including:

[0006] Acquire the environment image synchronously collected by the binocular camera;

[0007] Extracting a plurality of first feature points from a first environment image captured by a first camera;

[0008] For each first feature point, a depth range is set, and multiple depth values ​​are obtained within the depth range based on a preset step size, and according to each depth value, the grayscale difference between a point in a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by a second camera is calculated, and the depth value with the smallest grayscale difference is used as the initial depth of binocular matching;

[0009] According to the pre-calibrated positional relationship between the binocular cameras, the initial depth is linearly optimized to obtain a target depth of binocular matching;

[0010] According to the target depth, calculating the second feature point corresponding to each first feature point in the second environment image, and generating a set of feature point matching pairs;

[0011] SALM positioning is performed based on the matching pairs of feature points.

[0012] Optionally, the step of obtaining a plurality of depth values ​​within the depth range based on a preset step size, and calculating the grayscale difference between a point within a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by a second camera according to each depth value, and taking the depth value with the smallest grayscale difference as the initial depth of binocular matching, includes:

[0013] Taking the lower limit of the depth range as the current depth value;

[0014] For each current point within a preset neighborhood of the first feature point, according to the current depth value and in combination with the positional relationship between the binocular cameras, calculate a same-name point of the current point in the second environment image captured by the second camera, and calculate a grayscale difference between the current point and the same-name point;

[0015] Summing the grayscale differences corresponding to each point in a preset neighborhood of the first feature point to obtain the grayscale difference;

[0016] Increasing the current depth value by a preset step size, and recalculating the grayscale difference until the current depth value reaches an upper limit of the depth range;

[0017] If the minimum grayscale difference among the grayscale differences is smaller than the first preset threshold, the depth value corresponding to the minimum grayscale difference is used as the initial depth of the binocular matching.

[0018] Optionally, the calculation formula for the minimum grayscale difference corresponding to the initial depth is:

[0019]

[0020] Among them, N(x L ) is the first feature point x L The i-th point in the preset neighborhood of xRi is the same-name point of the i-th point in the second environment image, I L (x Li ) is the gray value of the i-th point in the first environment image, I R (x Ri ) is the gray value of the i-th point in the second environment image, d L is the depth value within the depth range.

[0021] Optionally, the linear optimization of the initial depth according to the pre-calibrated positional relationship between the binocular cameras to obtain the target depth of binocular matching includes:

[0022] Establishing a grayscale invariance equation according to the initial depth, the depth variation and the positional relationship, wherein the grayscale invariance equation introduces Gaussian noise to make the grayscale values ​​of the feature points with the same name in the first environment image and the second environment image equal;

[0023] Taking partial derivatives of the grayscale invariance equation, and obtaining a grayscale change linear equation based on points in a preset neighborhood of each first feature point and points with the same name;

[0024] The least square method is used to iterate the grayscale linear change equation multiple times to solve the target depth.

[0025] Optionally, the calculation formula of the second feature point is:

[0026]

[0027] Among them, π R (·) is the projection equation of the second camera, π L (·) is the projection equation of the first camera, is the positional relationship between the binocular cameras, d′ L is the target depth, x L is the first feature point in the first environment image captured by the first camera, x R The second feature point is a second feature point in the second environment image captured by the second camera and corresponds to the first feature point.

[0028] On the other hand, an embodiment of the present application provides a virtual reality device, including a processor, a memory, and at least one pair of binocular cameras, wherein the binocular cameras, the memory, and the processor are connected via a bus;

[0029] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0030] Acquire the environment image synchronously collected by the binocular camera;

[0031] Extracting a plurality of first feature points from a first environment image captured by a first camera;

[0032] For each first feature point, a depth range is set, and multiple depth values ​​are obtained within the depth range based on a preset step size, and according to each depth value, the grayscale difference between a point in a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by a second camera is calculated, and the depth value with the smallest grayscale difference is used as the initial depth of binocular matching;

[0033] According to the pre-calibrated positional relationship between the binocular cameras, the initial depth is linearly optimized to obtain a target depth of binocular matching;

[0034] According to the target depth, calculating the second feature point corresponding to each first feature point in the second environment image, and generating a set of feature point matching pairs;

[0035] SALM positioning is performed based on the matching pairs of feature points.

[0036] Optionally, the processor obtains a plurality of depth values ​​within the depth range based on a preset step size, and calculates, based on each depth value, a grayscale difference between a point within a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by the second camera, and uses a depth value with the smallest grayscale difference as an initial depth for binocular matching. The specific operation is:

[0037] Taking the lower limit of the depth range as the current depth value;

[0038] For each current point within a preset neighborhood of the first feature point, according to the current depth value and in combination with the positional relationship between the binocular cameras, calculate a same-name point of the current point in the second environment image captured by the second camera, and calculate a grayscale difference between the current point and the same-name point;

[0039] Summing the grayscale differences corresponding to each point in a preset neighborhood of the first feature point to obtain the grayscale difference;

[0040] Increasing the current depth value by a preset step size, and recalculating the grayscale difference until the current depth value reaches an upper limit of the depth range;

[0041] If the minimum grayscale difference among the grayscale differences is smaller than the first preset threshold, the depth value corresponding to the minimum grayscale difference is used as the initial depth of the binocular matching.

[0042] Optionally, the calculation formula for the minimum grayscale difference corresponding to the initial depth is:

[0043]

[0044] Among them, N(x L ) is the i-th point in the preset neighborhood of the first feature point xL, x Ri is the same-name point of the i-th point in the second environment image, I L (x Li ) is the gray value of the i-th point in the first environment image, I R (x Ri ) is the gray value of the i-th point in the second environment image, d L is the depth value within the depth range.

[0045] Optionally, the processor performs linear optimization on the initial depth according to a pre-calibrated positional relationship between the binocular cameras to obtain a target depth for binocular matching, and the specific operation is:

[0046] Establishing a grayscale invariance equation according to the initial depth, the depth variation and the positional relationship, wherein the grayscale invariance equation introduces Gaussian noise to make the grayscale values ​​of the feature points with the same name in the first environment image and the second environment image equal;

[0047] Taking partial derivatives of the grayscale invariance equation, and obtaining a grayscale change linear equation based on points in a preset neighborhood of each first feature point and points with the same name;

[0048] The least square method is used to iterate the grayscale linear change equation multiple times to solve the target depth.

[0049] Optionally, the calculation formula of the second feature point is:

[0050]

[0051] Among them, π R (·) is the projection equation of the second camera, π L (·) is the projection equation of the first camera, is the positional relationship between the binocular cameras, d′ L is the target depth, x L is the first feature point in the first environment image captured by the first camera, x R The second feature point is a second feature point in the second environment image captured by the second camera and corresponds to the first feature point.

[0052] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a binocular matching method for SLAM positioning provided in an embodiment of the present application.

[0053] The beneficial effects of a binocular matching method for SLAM positioning provided by an embodiment of the present application are as follows:

[0054] For each first feature point extracted from the first environment image captured by the first camera, based on the grayscale invariance assumption, a rough search is performed within a given depth range with a preset step size to query the initial depth that minimizes the grayscale difference between the points within the preset neighborhood of the first feature point and the points of the same name in the second environment image captured by the second camera, and through linear optimization, the precise target depth is obtained, and combined with the pre-calibrated external parameters that characterize the positional relationship between the binocular cameras, the feature point matching pairs of the binocular matching are accurately calculated, effectively overcoming the rotation and scale inconsistency between the feature points in the binocular image caused by the inconsistent orientation of the camera installation, and without the need to correct the image, which reduces the correction error while also reducing the resource consumption of image correction, thereby improving the speed of binocular matching, and thus ensuring real-time high-precision SLAM positioning results.

[0055] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 A schematic diagram of the binocular camera orientation provided in an embodiment of the present application;

[0058] Figure 2A A schematic diagram of a VR device provided in an embodiment of the present application with two pairs of binocular cameras installed;

[0059] Figure 2B A schematic diagram of an environmental image synchronously captured by four cameras provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of the overall algorithm framework of binocular matching based on SLAM positioning provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the grayscale invariance assumption principle provided in the embodiments of the present application;

[0062] Figure 5 A flowchart of a binocular matching method for SLAM positioning provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of a neighborhood of characteristic points provided in an embodiment of the present application;

[0064] Figure 7 A flow chart of a depth value rough search method provided in an embodiment of the present application;

[0065] Figure 8 A flow chart of a depth value precise search method provided in an embodiment of the present application;

[0066] Fig. 9 A structural diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0068] Feature matching is one of the main sources of mid-scale information in SLAM positioning. Inaccurate feature matching will directly affect the SLAM positioning results.

[0069] For VR devices, in order to avoid the loss of environmental features caused by camera occlusion, multiple cameras are set with large differences in orientation. Figure 2A As shown in the figure, it is a schematic diagram of VR equipment. The four cameras are circled. The environmental images collected by different cameras at the same time are as follows: Figure 2B As shown. Figure 2B It can be seen that the difference in orientation between cameras makes the rotation and scale of the captured images inconsistent. The same feature point will show different rotations and scales in the images captured by different cameras at the same time, which increases the difficulty of feature matching.

[0070] At present, the binocular matching methods in SLAM positioning mainly include the following:

[0071] (1) Binocular matching method based on photometric invariance assumption

[0072] The binocular matching method based on the assumption of photometric invariance (such as the optical flow method) is used to perform feature matching on images captured by the binocular camera. The matching speed is fast and the number of matched feature point pairs is large. However, this method itself does not have scale and rotation invariance, and cannot handle the matching of feature points with scale and rotation changes caused by different camera orientations, and the matching accuracy is poor.

[0073] (2) Descriptor-based binocular matching method

[0074] Use a descriptor-based binocular matching method (such as ORB feature matching) to perform feature matching on images captured by the binocular camera. Since the ORB feature itself has a certain degree of scale and rotation invariance, it can solve the matching of feature points with scale and rotation changes caused by different camera orientations. However, this method has a slow matching speed and a small number of matched feature point pairs, which cannot meet the real-time high-precision SLAM positioning requirements.

[0075] (3) Fusion binocular matching method

[0076] This method requires first correcting the images captured by the binocular camera to eliminate scale and rotation changes, and then using the first two methods for fusion feature matching. Although it can overcome the influence of scale and rotation changes on feature matching to a certain extent, image correction is time-consuming and has poor real-time performance.

[0077] In view of this, an embodiment of the present application provides a binocular matching method for SLAM positioning. The method performs a rough search within a given depth range with a preset step size for each first feature point extracted from a first environment image captured by a first camera based on the grayscale invariance assumption, and queries the initial depth that minimizes the grayscale difference between points within a preset neighborhood of the first feature point and the same-name points in a second environment image captured by a second camera, and obtains an accurate target depth through linear optimization, thereby accurately calculating the feature point pairs for binocular matching. The method can effectively overcome the rotation and scale inconsistencies caused by inconsistent camera installation orientations, and does not require image correction. While reducing the correction error, it also reduces the resource consumption of image correction, improves the speed of binocular matching, and thus ensures real-time, high-precision SLAM positioning results.

[0078] See also Figure 3, which is a schematic diagram of the overall algorithm framework of binocular matching based on SLAM positioning provided in an embodiment of the present application. First, feature points in the environment image captured by a monocular camera (which can be a left camera or a right camera) are extracted, and then a rough search of the depth value is performed within a preset neighborhood of each extracted feature point. If the rough search is successful, the depth value of the rough search is used as the initial depth, and a fine search of the depth value is performed through linear optimization. If the fine search is successful, the feature points that match each feature point in the environment image captured by another monocular camera are restored according to the target depth of the fine search, and feature point matching pairs with rotation and scale invariance are quickly obtained, thereby improving the real-time performance and accuracy of SLAM positioning.

[0079] Among them, in the process of coarse search and fine search of depth value, the grayscale invariance assumption principle is adopted, such as Figure 4 As shown in the figure, the positions of the same-name points collected by the binocular camera on the environment image are respectively denoted as x L 、x R , the projection equations of the binocular cameras are π L (·) and π R (·), the extrinsic parameters for calibration between binocular cameras are Represents the rotation relationship between binocular cameras, represents the translation relationship between the binocular cameras. Therefore, the external parameters can characterize the positional relationship between the binocular cameras. Assume that the depth value of the feature point in the left camera is d L , then the position of the same-name point in the right camera is:

[0080]

[0081] Similarly, assuming that the depth value of the feature point in the right camera is d R , then the position of the same-name point in the left camera is:

[0082]

[0083] Based on the grayscale invariance assumption principle, the process of the binocular matching method for SLAM positioning provided in the embodiment of the present application is as follows: Figure 5 As shown, the process is executed by the VR device, specifically the HMD in the VR device, and mainly includes the following steps:

[0084] S501: Acquire an environment image synchronously collected by a binocular camera.

[0085] In practical applications, the binocular cameras on VR devices have different orientations but the same field of view. Therefore, the environmental images around the VR device can be synchronously collected from different perspectives. The two environmental images contain the same-name points of the same object.

[0086] S502: Extracting a plurality of first feature points from a first environment image captured by a first camera.

[0087] In order to avoid the influence of different camera orientations on feature point matching, in one example, only the environment image captured by the monocular camera (i.e., the first camera) is used for feature extraction, and the feature points in the environment image captured by the other camera (i.e., the second camera) can be solved by the grayscale invariance assumption principle. The first camera can be a left camera or a right camera. When the first camera is a left camera, the second camera is a right camera, and when the first camera is a right camera, the second camera is a left camera.

[0088] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the algorithm for extracting feature points in the environment image captured by the first camera, including but not limited to the ORB algorithm, SIFT algorithm or SURF algorithm.

[0089] S503: For each first feature point, set a depth range, obtain multiple depth values ​​within the depth range based on a preset step size, and calculate the grayscale difference between the points in a preset neighborhood of the first feature point and the points of the same name in the second environment image captured by the second camera based on each depth value, and use the depth value with the smallest grayscale difference as the initial depth of binocular matching.

[0090] In one example, for each first feature point extracted from the first environment image, such as Figure 6 As shown, at the first feature point x L (exist Figure 6 In the preset neighborhood (indicated by a black square in the figure), there is a L The neighboring points (in Figure 6 These points in this neighborhood may belong to the same object, so their depth values ​​(indicating the distance from the object to the camera) are the same, and based on the grayscale invariance assumption, the grayscale values ​​of the points in this neighborhood are the same as those of the same points in the second environment image captured by the second camera. Therefore, a depth range [d min , d max ], the depth value d∈[d min , d max ]. Within this depth range, based on the grayscale invariance assumption, a coarse search of the depth value is performed with a preset step size to obtain an initial depth that minimizes the grayscale difference between points within a preset neighborhood of each first feature point and the same-name points in the second environment image.

[0091] The depth value rough search process can be found in Figure 7 , mainly includes the following steps:

[0092] S5031: Taking the lower limit of the depth range as the current depth value.

[0093] S5032: For each current point within the preset neighborhood of the first feature point, based on the current depth value and in combination with the positional relationship between the binocular cameras, calculate the same-name point of the current point in the second environment image captured by the second camera, and calculate the grayscale difference between the current point and the same-name point.

[0094] For each current point within the preset neighborhood of the first feature point, Formula 1 (the first camera is the left camera) or Formula 2 (the first camera is the right camera) can be used to calculate the same-name point in the second environment image captured by the second camera, and the grayscale value of the current point is obtained from the first environment image, and the grayscale value of the same-name point is obtained from the second environment image, thereby calculating the grayscale difference between the two.

[0095] S5033: Sum the grayscale differences corresponding to each point in a preset neighborhood of the first feature point to obtain a grayscale difference.

[0096] Specifically, the formula for grayscale difference is expressed as:

[0097]

[0098] Among them, x L represents the first feature point, N(x L ) is the first feature point x L The preset neighborhood of x Li is the i-th point in the preset neighborhood, x Ri is the same-name point of the i-th point in the second environment image, I L (x Li ) is the gray value of the i-th point in the first environment image, I R (x Ri ) is the gray value of the point with the same name as the i-th point in the second environment image.

[0099] S5034: Increase the current depth value by a preset step size to obtain a new current depth value.

[0100] Among them, the size of the preset step size can be set according to the depth range size of each first feature point, and the embodiment of the present application does not make any restrictive requirements.

[0101] S5035: Determine whether the current depth value exceeds the upper limit of the depth range, if so, execute S5036, if not, return to S5032.

[0102] When the increased depth value is equal to the upper limit of the depth range, it indicates that all depth values ​​in the depth range have been traversed and the depth value search can be ended; otherwise, the depth value search needs to be continued.

[0103] S5036: Sort the grayscale differences and select the minimum grayscale difference.

[0104] Among them, the selection formula of grayscale difference is as follows:

[0105]

[0106] S5037: Determine whether the minimum grayscale difference is less than a first preset threshold, if so, execute S5038, if not, return to S501.

[0107] Assume that the first preset threshold is th coarse ,like It indicates that the coarse search of the depth value is successful, otherwise, it indicates that the coarse search of the depth value fails.

[0108] S5038: Using the depth value corresponding to the minimum grayscale difference as the initial depth for binocular matching.

[0109] When the coarse search of the depth value is successful, the fine search of the depth value can be realized by linearly optimizing the initial depth, thereby improving the accuracy of the calculation of the same-name points in the second environment image and improving the binocular matching accuracy.

[0110] S504: According to the pre-calibrated positional relationship between the binocular cameras, linear optimization is performed on the initial depth to obtain a target depth for binocular matching.

[0111] In practical applications, the positional relationship between binocular cameras can be obtained through pre-calibrated extrinsic parameters. Combined with the initial depth of the rough search, based on the grayscale invariance assumption, the grayscale equation between the corresponding points in the two images can be established, thereby obtaining accurate depth values ​​through linear optimization.

[0112] The process of depth value precise search can be found in Figure 8 , mainly includes the following steps:

[0113] S5041: Establish a grayscale invariance equation based on the initial depth, depth change and position relationship.

[0114] Among them, the grayscale invariance equation introduces Gaussian noise to make the grayscale values ​​of the feature points in the first environment image and the second environment image equal, and the formula is expressed as follows:

[0115]

[0116] Where n is zero-mean Gaussian noise, d L is the initial depth, and Δd is the depth change.

[0117] S5042: Calculate the partial derivative of the grayscale invariance equation, and obtain the grayscale linear change equation based on the points in the preset neighborhood of each first feature point and the points with the same name.

[0118] Among them, the result of partial derivative of formula 5 is:

[0119]

[0120] From formula 6, we can see that for each point in the preset neighborhood of the first feature point, the value of L +Δd) to determine the same-name point in the second environment image, the grayscale linear change equation can be established:

[0121]

[0122] S5043: Using the least square method, the grayscale change equation is iterated multiple times, and each iteration calculates the grayscale difference between the points in the preset neighborhood of each first feature point and the points with the same name in the second environment image.

[0123] The depth change can be obtained by formula 7:

[0124]

[0125] The points in the preset neighborhood of each first feature point and its corresponding points of the same name are iteratively updated using the least squares method. L =d L +Δd, and thus the grayscale difference is calculated by Formula 3.

[0126] S5044: Determine whether the grayscale difference calculated in each iteration is less than a second preset threshold, if so, execute S5045, if not, return to S501.

[0127] Assume that the second preset threshold is th fine , if the currently calculated It indicates that the model converges and the iteration can be ended, and the depth value fine search is successful. Otherwise, it indicates that the depth value fine search fails.

[0128] S5045: Taking the depth value corresponding to the grayscale difference smaller than the second preset threshold as the target depth.

[0129] S505: Calculate, according to the target depth, the second feature point corresponding to each first feature point in the second environment image, and generate a set of feature point matching pairs.

[0130] After obtaining the target depth, the second feature point corresponding to each first feature point can be obtained by substituting into the following formula.

[0131]

[0132] Among them, x L represents the first feature point in the first environment image captured by the first camera, xR represents the second feature point in the second environment image captured by the second camera, d′ L is the target depth, (x L , x R ) is the feature point matching pair of binocular matching.

[0133] S506: Perform SALM positioning according to the matching pairs of feature points.

[0134] After obtaining the matching pairs of feature points in the first environment image and the second environment image, the PnP algorithm is used to calculate the initial 6Dof pose. Combined with the motion measurement data collected by the IMU, the Kalman filter is used to optimize the initial 6Dof pose to obtain the accurate target 6Dof pose.

[0135] It should be noted that SLAM positioning is a relatively mature algorithm and will not be described in detail here.

[0136] The binocular matching method for SLAM positioning provided in the embodiment of the present application is based on the principle of grayscale invariance, and performs a coarse search and a fine search of the depth value for the preset neighborhood points of each first feature point collected by the monocular camera, so as to obtain the target depth that accurately characterizes the distance from the same object to the camera in the field of view of the binocular camera, and then combines the external parameters between the pre-calibrated binocular cameras, and uses the projection relationship of the binocular cameras to determine the feature points in the monocular image corresponding to the feature points in the other image, so as to achieve SLAM positioning. This method can solve the rotation and scale differences between the feature points in the binocular image caused by different camera orientations, obtain accurate binocular matching results, and there is no need to correct the binocular image. The original binocular image can be directly used for matching, thereby effectively reducing the interpolation error caused by image correction and the computing resources consumed by image correction, improving the efficiency of binocular matching, and meeting the real-time positioning requirements.

[0137] Based on the same technical concept, an embodiment of the present application provides a virtual reality device, which can implement the steps of the above-mentioned binocular matching method for SLAM positioning and achieve the same technical effect.

[0138] See also Fig. 9 , the virtual reality device includes a processor 901, a memory 902 and at least one pair of binocular cameras 903, wherein the binocular cameras 903, the memory 902 and the processor 901 are connected via a bus 904;

[0139] The memory 902 stores a computer program, and the processor 901 performs the following operations according to the computer program:

[0140] Acquire the environment image synchronously collected by the binocular camera 903;

[0141] Extracting a plurality of first feature points from a first environment image captured by a first camera;

[0142] For each first feature point, a depth range is set, and multiple depth values ​​are obtained within the depth range based on a preset step size, and according to each depth value, the grayscale difference between a point in a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by a second camera is calculated, and the depth value with the smallest grayscale difference is used as the initial depth of binocular matching;

[0143] According to the pre-calibrated positional relationship between the binocular cameras, the initial depth is linearly optimized to obtain a target depth of binocular matching;

[0144] According to the target depth, calculating the second feature point corresponding to each first feature point in the second environment image, and generating a set of feature point matching pairs;

[0145] SALM positioning is performed based on the matching pairs of feature points.

[0146] Optionally, the processor 901 obtains a plurality of depth values ​​within the depth range based on a preset step size, and calculates, based on each depth value, a grayscale difference between a point within a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by the second camera, and uses a depth value with the smallest grayscale difference as an initial depth for binocular matching. The specific operation is as follows:

[0147] Taking the lower limit of the depth range as the current depth value;

[0148] For each current point within a preset neighborhood of the first feature point, according to the current depth value and in combination with the positional relationship between the binocular cameras, calculate a same-name point of the current point in the second environment image captured by the second camera, and calculate a grayscale difference between the current point and the same-name point;

[0149] Summing the grayscale differences corresponding to each point in a preset neighborhood of the first feature point to obtain the grayscale difference;

[0150] Increasing the current depth value by a preset step size, and recalculating the grayscale difference until the current depth value reaches an upper limit of the depth range;

[0151] If the minimum grayscale difference among the grayscale differences is smaller than the first preset threshold, the depth value corresponding to the minimum grayscale difference is used as the initial depth of the binocular matching.

[0152] Optionally, the calculation formula for the minimum grayscale difference corresponding to the initial depth is:

[0153]

[0154] Among them, N(x L ) is the first feature point x L The i-th point in the preset neighborhood of x Ri is the same-name point of the i-th point in the second environment image, I L (x Li ) is the gray value of the i-th point in the first environment image, I R (x Ri ) is the gray value of the i-th point in the second environment image, d L is the depth value within the depth range.

[0155] Optionally, the processor 901 performs linear optimization on the initial depth according to the pre-calibrated positional relationship between the binocular cameras to obtain a target depth for binocular matching, and the specific operation is:

[0156] Establishing a grayscale invariance equation according to the initial depth, the depth variation and the positional relationship, wherein the grayscale invariance equation introduces Gaussian noise to make the grayscale values ​​of the feature points with the same name in the first environment image and the second environment image equal;

[0157] Taking partial derivatives of the grayscale invariance equation, and obtaining a grayscale change linear equation based on points in a preset neighborhood of each first feature point and points with the same name;

[0158] The least square method is used to iterate the grayscale linear change equation multiple times to solve the target depth.

[0159] Optionally, the calculation formula of the second feature point is:

[0160]

[0161] Among them, π R (·) is the projection equation of the second camera, π L (·) is the projection equation of the first camera, is the positional relationship between the binocular cameras, d′ L is the target depth, x L is the first feature point in the first environment image captured by the first camera, x R The second feature point is a second feature point in the second environment image captured by the second camera and corresponds to the first feature point.

[0162] in, Fig. 9The memory 902 in the embodiment may be a volatile memory, such as a random-access memory (RAM); the memory 902 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 902 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor 901 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0163] It should be noted that Fig. 9 This is just an example, and provides the necessary hardware for the virtual reality device to execute the steps of the binocular matching method for SLAM positioning provided in the embodiment of the present application. If not shown, the virtual reality device may also include the hardware of conventional VR devices such as speakers, microphones, function buttons, power supplies, communication interfaces, IMUs, and display screens.

[0164] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, a binocular matching method for SLAM positioning in the aforementioned embodiment can be completed.

[0165] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a binocular matching method for SLAM positioning in the aforementioned embodiment.

[0166] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0167] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0170] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A binocular matching method for SLAM positioning, characterized in that: Applied to virtual reality devices, including: Acquire the environment image collected synchronously by the binocular camera; Extracting a plurality of first feature points from a first environment image captured by a first camera; For each first feature point, a depth range is set, and multiple depth values ​​are obtained within the depth range based on a preset step size, and according to each depth value, the grayscale difference between a point in a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by a second camera is calculated, and the depth value with the smallest grayscale difference is used as the initial depth of binocular matching; According to the pre-calibrated positional relationship between the binocular cameras, the initial depth is linearly optimized to obtain a target depth of binocular matching; According to the target depth, calculating the second feature point corresponding to each first feature point in the second environment image, and generating a set of feature point matching pairs; SALM positioning is performed based on the matching pairs of feature points.

2. The method according to claim 1, characterized in that The method of obtaining a plurality of depth values ​​based on a preset step length within the depth range, and calculating the grayscale difference between a point within a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by a second camera according to each depth value, and taking the depth value with the smallest grayscale difference as the initial depth of binocular matching, includes: Taking the lower limit of the depth range as the current depth value; For each current point within a preset neighborhood of the first feature point, according to the current depth value and in combination with the positional relationship between the binocular cameras, calculate a same-name point of the current point in the second environment image captured by the second camera, and calculate a grayscale difference between the current point and the same-name point; Summing the grayscale differences corresponding to each point in a preset neighborhood of the first feature point to obtain the grayscale difference; Increasing the current depth value by a preset step size, and recalculating the grayscale difference until the current depth value reaches an upper limit of the depth range; If the minimum grayscale difference among the grayscale differences is smaller than the first preset threshold, the depth value corresponding to the minimum grayscale difference is used as the initial depth of the binocular matching.

3. The method according to claim 2, characterized in that The calculation formula of the minimum grayscale difference corresponding to the initial depth is: Among them, N(x L ) is the first feature point x L The i-th point in the preset neighborhood of x Ri is the same-name point of the i-th point in the second environment image, I L (x Li ) is the gray value of the i-th point in the first environment image, I R (x Ri ) is the gray value of the i-th point in the second environment image, d L is the depth value within the depth range.

4. The method according to claim 1, characterized in that The linear optimization of the initial depth according to the pre-calibrated positional relationship between the binocular cameras to obtain the target depth of binocular matching includes: Establishing a grayscale invariance equation according to the initial depth, the depth variation and the positional relationship, wherein the grayscale invariance equation introduces Gaussian noise to make the grayscale values ​​of the feature points with the same name in the first environment image and the second environment image equal; Taking partial derivatives of the grayscale invariance equation, and obtaining a grayscale change linear equation based on points in a preset neighborhood of each first feature point and points with the same name; The least square method is used to iterate the grayscale linear change equation multiple times to solve the target depth.

5. The method according to claim 1, characterized in that The calculation formula of the second feature point is: Among them, π R (·) is the projection equation of the second camera, π L (·) is the projection equation of the first camera, is the positional relationship between the binocular cameras, d′ L is the target depth, x L is the first feature point in the first environment image captured by the first camera, x R The second feature point is a second feature point in the second environment image captured by the second camera and corresponds to the first feature point.

6. A virtual reality device, characterized in that: It includes a processor, a memory and at least one pair of binocular cameras, wherein the binocular cameras, the memory and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquire the environment image synchronously collected by the binocular camera; Extracting a plurality of first feature points from a first environment image captured by a first camera; For each first feature point, a depth range is set, and multiple depth values ​​are obtained within the depth range based on a preset step size, and according to each depth value, the grayscale difference between a point in a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by a second camera is calculated, and the depth value with the smallest grayscale difference is used as the initial depth of binocular matching; According to the pre-calibrated positional relationship between the binocular cameras, the initial depth is linearly optimized to obtain a target depth of binocular matching; According to the target depth, calculating the second feature point corresponding to each first feature point in the second environment image, and generating a set of feature point matching pairs; SALM positioning is performed based on the matching pairs of feature points.

7. The virtual reality device according to claim 6, characterized in that: The processor obtains a plurality of depth values ​​within the depth range based on a preset step size, and calculates the grayscale difference between a point within a preset neighborhood of the first feature point and a point of the same name in a second environment image captured by the second camera according to each depth value, and uses the depth value with the smallest grayscale difference as the initial depth of binocular matching, and the specific operation is as follows: Taking the lower limit of the depth range as the current depth value; For each current point within a preset neighborhood of the first feature point, according to the current depth value and in combination with the positional relationship between the binocular cameras, calculate a same-name point of the current point in the second environment image captured by the second camera, and calculate a grayscale difference between the current point and the same-name point; Summing the grayscale differences corresponding to each point in a preset neighborhood of the first feature point to obtain the grayscale difference; Increasing the current depth value by a preset step size, and recalculating the grayscale difference until the current depth value reaches an upper limit of the depth range; If the minimum grayscale difference among the grayscale differences is smaller than the first preset threshold, the depth value corresponding to the minimum grayscale difference is used as the initial depth of the binocular matching.

8. The virtual reality device according to claim 7, characterized in that: The calculation formula of the minimum grayscale difference corresponding to the initial depth is: Among them, N(x L ) is the first feature point x L The i-th point in the preset neighborhood of x Ri is the same-name point of the i-th point in the second environment image, I L (x Li ) is the gray value of the i-th point in the first environment image, I R (x Ri ) is the gray value of the i-th point in the second environment image, d L is the depth value within the depth range.

9. The virtual reality device according to claim 6, characterized in that: The processor performs linear optimization on the initial depth according to the pre-calibrated positional relationship between the binocular cameras to obtain the target depth of binocular matching, and the specific operation is: Establishing a grayscale invariance equation according to the initial depth, the depth variation and the positional relationship, wherein the grayscale invariance equation introduces Gaussian noise to make the grayscale values ​​of the feature points with the same name in the first environment image and the second environment image equal; Taking partial derivatives of the grayscale invariance equation, and obtaining a grayscale change linear equation based on points in a preset neighborhood of each first feature point and points with the same name; The least square method is used to iterate the grayscale linear change equation multiple times to solve the target depth.

10. The virtual reality device according to claim 6, characterized in that: The calculation formula of the second feature point is: Among them, π R (·) is the projection equation of the second camera, π L (·) is the projection equation of the first camera, is the positional relationship between the binocular cameras, d′ L is the target depth, x L is the first feature point in the first environment image captured by the first camera, x R The second feature point is a second feature point in the second environment image captured by the second camera and corresponds to the first feature point.

Images