A parallax automatic correction method, device, equipment and readable storage medium

By adjusting the displacement and matching the view of the binocular shooting device and determining the working distance based on the homography matrix, automatic parallax correction is achieved, solving the problems of convergence adjustment conflicts caused by binocular parallax and high calculation costs, and improving Comfort and accuracy of stereo vision.

CN117615113BActive Publication Date: 2024-11-12ZHEJIANG HEALNOC TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311318929.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-11
Publication Date
2024-11-12
Estimated Expiration
2043-10-11

AI Technical Summary

Technical Problem

In the existing stereo vision technology, the convergence adjustment conflict caused by binocular parallax leads to the observer fatigue and dizziness, and the existing parallax correction method has high calculation cost and low accuracy, and relies on hardware ranging unit to increase the equipment volume and cost.

Method used

By adjusting the displacement of the left and right views of the binocular shooting device, a feature point pair and pixel point pair are obtained, the current working distance is determined based on the homography matrix, and the view displacement is adjusted using the image displacement table to achieve automatic parallax correction.

Benefits of technology

It reduces the calculation cost of the automatic parallax correction process, improves the accuracy and stability of depth information, reduces imaging parallax, and improves the comfort of stereoscopic visual display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117615113B_ABST
    Figure CN117615113B_ABST
Patent Text Reader

Abstract

The present application relates to a method, apparatus, device and readable storage medium for automatic parallax correction, wherein the method for automatic parallax correction comprises: respectively performing displacement adjustment on a left view and a right view acquired by a binocular shooting device at different working distances to obtain an image displacement table; obtaining a plurality of feature point pairs in the left view and the right view at a current working distance; determining all corresponding pixel point pairs in the left view and the right view based on a coordinate mapping relationship between the plurality of feature point pairs; determining a current working distance of the binocular shooting device based on the coordinates of all corresponding pixel point pairs; determining a displacement adjustment value of the left view and a displacement adjustment value of the right view based on the current working distance of the binocular shooting device and the image displacement table, so as to respectively perform movement adjustment on the left view and the right view, thereby reducing the computational cost consumption in the process of automatic parallax correction and improving the accuracy of automatic parallax correction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of stereoscopic imaging technology, and in particular to a method, device, equipment and readable storage medium for automatic parallax correction. Background Art

[0002] With the rapid development of technologies such as virtual reality, stereoscopic vision is becoming a widely-watched technology, and its application in different fields is expanding rapidly. Currently, the most common stereoscopic vision technology is based on binocular parallax. The observer's eyes obtain a two-dimensional image of the same object from different directions. There is a difference between the left and right images. The brain analyzes and processes this difference to obtain information such as the brightness, shape, and spatial distribution of the object to distinguish the distance of the object, thereby creating a sense of three-dimensionality.

[0003] Stereoscopic vision technology also plays an important role in the medical field. Compared with traditional technologies, stereoscopic vision technology can provide three-dimensional reconstruction of surgical scenes for minimally invasive surgery, enrich the depth information of the scene, and improve the safety of surgical operations. However, in actual use, because the eyes are focused on the display screen, when the user observes objects at different depths, the display image and the inferred depth information of the visual system do not match, which is prone to vergence-accommodation conflict (VAC). Long-term use will cause fatigue and dizziness. In stereoscopic imaging, the main factor affecting the observer's perception of depth is binocular parallax. Therefore, we can adjust the parallax to a comfortable range according to the distribution range of the parallax, thereby improving the comfort of stereoscopic vision.

[0004] In the prior art, a stereo matching algorithm (such as BM (Block Matching) algorithm, SGBM (Semi-Global Block Matching) algorithm, etc.) is usually used to obtain a depth image. It is necessary to set a fixed window to slide the image and use a cost function as an evaluation to complete the matching of image blocks in the two images. Since the stereo matching algorithm is sensitive to parameters, the size of the window has a greater impact on the matching result. The smaller the window, the greater the noise of the obtained depth map. The larger the window, the more likely it is to cause mismatching, which is reflected in the increase of holes in the depth map. Therefore, it has the problem of high computational cost and low accuracy. Alternatively, the distance from the photographed object to the camera is obtained by using ranging hardware units such as infrared ranging, laser ranging or ultrasonic ranging; then the disparity value between the left and right images is obtained according to the distance of the photographed object, but it has great limitations in use, and the addition of ranging hardware will not only affect the size of the device, but also increase the cost. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, equipment and readable storage medium for automatic parallax correction in order to solve the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides a method for automatic parallax correction, the method comprising:

[0007] The left view and the right view obtained by the binocular shooting device at different working distances are respectively adjusted in displacement until the target stereoscopic vision effect is achieved, so as to obtain an image displacement table;

[0008] Get multiple feature point pairs in the left view and the right view at the current working distance;

[0009] Based on the coordinate mapping relationship between the multiple feature point pairs, determine all corresponding pixel point pairs in the left view and the right view;

[0010] Determining the current working distance of the binocular camera device based on the coordinates of all corresponding pixel pairs;

[0011] Based on the current working distance of the binocular shooting device and the image displacement table, the displacement adjustment value of the left view and the displacement adjustment value of the right view are determined to move and adjust the left view and the right view respectively.

[0012] In one embodiment, acquiring multiple feature point pairs in the left view and the right view at the current working distance includes:

[0013] Selecting a first region of interest in the left view, and selecting a second region of interest in the right view;

[0014] Using a feature point detection algorithm to perform feature point detection on the first region of interest and the second region of interest respectively to obtain corresponding feature points;

[0015] The feature points of the first region of interest are matched with the feature points of the second region of interest using a feature point matching algorithm to obtain a plurality of feature point pairs in the left view and the right view.

[0016] In one of the embodiments, determining all corresponding pixel pairs in the left view and the right view based on the coordinate mapping relationship between the coordinates of the plurality of feature point pairs comprises:

[0017] Solving a homography matrix based on the coordinates of the plurality of feature point pairs;

[0018] Based on the homography matrix, all corresponding pixel pairs in the left view and the right view are determined.

[0019] In one embodiment, solving the homography matrix based on the coordinates of the plurality of feature point pairs comprises:

[0020] Eliminate feature point pairs with incorrect matching among the multiple feature point pairs;

[0021] The homography matrix is ​​solved based on the coordinates of multiple feature point pairs after eliminating matching errors.

[0022] In one of the embodiments, the calculation formula for determining all corresponding pixel pairs in the left view and the right view based on the homography matrix is ​​specifically as follows:

[0023]

[0024] Where H is the homography matrix, is the coordinate of the pixel point in the left view, is the coordinate of the pixel point in the right view corresponding to the left view.

[0025] In one embodiment, obtaining the displacement adjustment value of the left view and the displacement adjustment value of the right view based on the current working distance of the binocular shooting device and the image displacement table includes:

[0026] Querying the image displacement table based on the current working distance of the binocular shooting device to obtain a displacement query result;

[0027] The displacement query result is interpolated to obtain a displacement adjustment value of the left view and a displacement adjustment value of the right view.

[0028] In one embodiment, querying the image displacement table based on the current working distance of the binocular shooting device to obtain a displacement query result; interpolating the displacement query result to obtain the displacement adjustment value of the left view and the displacement adjustment value of the right view includes:

[0029] Based on the current working distance of the binocular shooting device, determining in the image displacement table a first working distance and a second working distance adjacent to the current working distance;

[0030] Querying the image displacement table to determine a first displacement adjustment value corresponding to the first working distance and a second displacement adjustment value corresponding to the second working distance;

[0031] Interpolation calculation is performed on the first displacement adjustment value and the second displacement adjustment value to obtain the displacement adjustment value of the left view and the displacement adjustment value of the right view.

[0032] In a second aspect, an embodiment of the present application further provides a parallax automatic correction device, the device comprising:

[0033] The first acquisition module is used to respectively adjust the displacement of the left view and the right view obtained by the binocular shooting device at different working distances until the target stereoscopic vision effect is achieved, so as to obtain an image displacement table;

[0034] The second acquisition module is used to obtain multiple feature point pairs in the left view and the right view at the current working distance;

[0035] A determination module, configured to determine all corresponding pixel point pairs in the left view and the right view based on the coordinate mapping relationship between the plurality of feature point pairs;

[0036] A third obtaining module is used to determine the current working distance of the binocular shooting device based on the coordinates of all corresponding pixel pairs;

[0037] An adjustment module is used to determine the displacement adjustment value of the left view and the displacement adjustment value of the right view based on the current working distance of the binocular shooting device and the image displacement table, so as to move and adjust the left view and the right view respectively.

[0038] In a third aspect, an embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect above is implemented.

[0039] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0040] The above-mentioned parallax automatic correction method, device, equipment and readable storage medium are respectively adjusted by displacement of the left view and the right view obtained by the binocular shooting device at different working distances until the target stereoscopic visual effect is achieved to obtain an image displacement table; obtain multiple feature point pairs in the left view and the right view at the current working distance; based on the coordinate mapping relationship between the multiple feature point pairs, determine all corresponding pixel pairs in the left view and the right view; based on the coordinates of all corresponding pixel pairs, determine the current working distance of the binocular shooting device; based on the current working distance of the binocular shooting device and the image displacement table, determine the displacement adjustment value of the left view and the displacement adjustment value of the right view, so as to move and adjust the left view and the right view respectively. The present application can obtain the depth information of the image without relying on the hardware unit, and improves the method of obtaining the depth information, has better stability, and obtains the working distance of the current camera as accurately as possible to achieve the adjustment of the left and right views, reduce the imaging parallax, and improve the comfort of stereoscopic visual display. Compared with the existing method, the present application reduces the computational cost consumption in the process of automatic parallax correction, improves the stability and the accuracy of the obtained depth information.

[0041] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 is a schematic diagram of an application environment of a parallax automatic correction method in an embodiment;

[0044] Figure 2 is a schematic diagram of a flow chart of a parallax automatic correction method in an embodiment;

[0045] Figure 3 is a schematic diagram of a stereoscopic visual effect before image displacement adjustment according to an embodiment;

[0046] Figure 4 is a schematic diagram of a target stereoscopic visual effect after image displacement adjustment in an embodiment;

[0047] Figure 5 is a flowchart of executing specific steps of S202 in an embodiment;

[0048] Figure 6 is a structural block diagram of a parallax automatic correction method device in an embodiment;

[0049] Figure 7 It is a schematic diagram of the structure of a computer device in an embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0051] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0052] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the parallax automatic correction method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0055] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the parallax automatic correction method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0056] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Card, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0057] The present application embodiment provides a method for automatic parallax correction, such as Figure 2 As shown, the method comprises the following steps:

[0058] Step S201, respectively performing displacement adjustment on the left view and the right view acquired by the binocular shooting device at different working distances until the target stereoscopic visual effect is achieved, so as to obtain an image displacement table;

[0059] Specifically, before the binocular camera device works, the camera needs to be calibrated. The commonly used Zhang Zhengyou calibration method can be used to obtain the left and right views in the same coordinate plane. If two parallel left and right cameras are used, image acquisition can be performed directly.

[0060] This embodiment conducts a comfort experiment to obtain an image displacement table. First, the actual working distance of the binocular camera used is divided into multiple levels, and the number of divided levels can be increased within the frequently used local working distance range, and then the stereoscopic visual effect at each distance level is observed. In one example, Figure 3 The figure shows a schematic diagram of the three-dimensional effect of an object, in which: Figure 3 The left side shows the position of the object in the left view in the final display effect. Figure 3The middle right side indicates the position of the object in the right view in the final display effect. Due to the existence of parallax, the two do not overlap, so it is necessary to adjust the displacement of the left view and the right view respectively until a more comfortable viewing effect is achieved. For example Figure 4 The target stereoscopic visual effect is displayed, the same object in the left view and the right view overlaps in the final imaging, and the displacement value of the left view and the displacement value of the right view under the corresponding working distance are recorded. The image displacement table of this embodiment can limit the displacement adjustment of the image within a controllable range, and can take a smaller span between the frequently used working distance levels. The design of the non-uniform lookup table is adopted to ensure the accuracy of the displacement adjustment of the left and right views, while reducing the workload of the preliminary experiments.

[0061] Step S202, obtaining multiple feature point pairs in the left view and the right view at the current working distance;

[0062] Step S203, determining all corresponding pixel pairs in the left view and the right view based on the coordinate mapping relationship between the multiple feature point pairs;

[0063] Step S204, determining the current working distance of the binocular shooting device based on the coordinates of all corresponding pixel pairs;

[0064] Step S205 , based on the current working distance of the binocular shooting device and the image displacement table, determine the displacement adjustment value of the left view and the displacement adjustment value of the right view, so as to perform movement adjustment on the left view and the right view respectively.

[0065] In the above steps S201 to S205, the left view and the right view obtained by the binocular shooting device at different working distances are respectively adjusted for displacement until the target stereoscopic visual effect is achieved, so as to obtain an image displacement table; multiple feature point pairs in the left view and the right view at the current working distance are obtained; based on the coordinate mapping relationship between the multiple feature point pairs, all corresponding pixel pairs in the left view and the right view are determined; based on the coordinates of all corresponding pixel pairs, the current working distance of the binocular shooting device is determined; based on the current working distance of the binocular shooting device and the image displacement table, the displacement adjustment value of the left view and the displacement adjustment value of the right view are determined to move and adjust the left view and the right view respectively. The present application can obtain the depth information of the image without relying on the hardware unit, and improves the method for obtaining the depth information, has better stability, and obtains the working distance of the current camera as accurately as possible to achieve the adjustment of the left and right views, reduce the imaging parallax, and improve the comfort of the stereoscopic visual display. Compared with the existing method, the present application reduces the computational cost consumption in the process of automatic parallax correction, improves the stability and the accuracy of the obtained depth information.

[0066] In one embodiment, if Figure 5 As shown, the acquisition of multiple feature point pairs in the left view and the right view at the current working distance includes the following steps:

[0067] Step S301, selecting a first region of interest in the left view, and selecting a second region of interest in the right view;

[0068] Specifically, a first region of interest is selected in the left view, and a second region of interest is selected in the right view, wherein the first region of interest and the second region of interest have an overlapping area in the image, and the overlapping area is the region of interest.

[0069] Step S302, performing feature point detection on the first region of interest and the second region of interest respectively using a feature point detection algorithm to obtain corresponding feature points;

[0070] There are always unique pixel points in an image, and these points can be considered as features of the image, and become feature points. In this embodiment, a feature point detection algorithm is used to perform feature point detection on the first region of interest to obtain feature points of the region of interest in the left view, and a feature point detection algorithm is used to perform feature point detection on the second region of interest to obtain feature points of the region of interest in the right view. The feature point detection algorithm can use SIFT, SURF, FAST or ORB algorithm or other detection algorithms, which are not limited in this embodiment.

[0071] Among them, SURF (Speeded Up Robust Features) algorithm is an accelerated version of SIFT algorithm, while SIFT (Scalelnvariant Feature Transform) is another well-known scale-invariant feature detection method. Compared with SIFT, SURF has greatly improved the speed of feature point detection, so it has strong applications in some real-time video stream object matching. SIFT, however, has a huge amount of feature calculations, which makes the feature point extraction process extremely time-consuming, so it is difficult to be used in some speed-oriented occasions. However, the advantage of SIFT over SURF is that, since SIFT calculates feature points based on floating-point kernels, it is generally believed that the features detected by the SIFT algorithm are more accurately positioned in space and scale, so the SIFT algorithm can be considered in situations where extremely accurate matching is required and matching speed is not considered.

[0072] The FAST (Features from Accelerated Segment Test) algorithm is designed to quickly detect points of interest. It only needs to compare a few pixels to determine whether it is a key point. The FAST algorithm detects points of interest very quickly, so it is very suitable for applications that need to prioritize speed. These applications include real-time visual tracking, target recognition, etc., which require tracking or matching multiple points in a real-time video stream.

[0073] The ORB algorithm (Oriented Fast and Rotated Brief) is an improved version of the Brief algorithm. The ORB algorithm is faster than the SIFT algorithm and the SURF algorithm. In the field of computer vision, the comprehensive performance of the ORB algorithm is better than other feature extraction algorithms in various evaluations.

[0074] Step S303: Match the feature points of the first region of interest with the feature points of the second region of interest using a feature point matching algorithm to obtain a plurality of feature point pairs in the left view and the right view.

[0075] The feature point matching algorithm may be a brute force matching algorithm, a cross matching algorithm, a K nearest neighbor matching algorithm or other matching algorithms, which are not limited in this embodiment. This embodiment uses a feature point matching algorithm to obtain multiple feature point pairs in the left view and the right view.

[0076] In one embodiment, the determining all corresponding pixel pairs in the left view and the right view based on the coordinate mapping relationship between the coordinates of the plurality of feature point pairs includes: solving a homography matrix based on the coordinates of the plurality of feature point pairs; wherein there are many ways to solve the homography matrix, and the obtained homography matrix represents the conversion relationship between the coordinates of the left view and the right view. Therefore, all corresponding pixel pairs in the left view and the right view can be determined based on the homography matrix.

[0077] In the case where there may be mismatches in the multiple feature point pairs in the left view and the right view obtained in step S304, in one embodiment, a random sample consensus algorithm (SANSAC) is used to solve the homography matrix. The random sample consensus algorithm can robustly evaluate the homography matrix even when there are erroneous feature point pairs.

[0078] Specifically, firstly, the feature point pairs with matching errors are eliminated from the plurality of feature point pairs using a random sampling consensus algorithm; based on the coordinates of the plurality of feature point pairs after eliminating the matching errors, the random sampling consensus algorithm solves the homography matrix. In this embodiment, other algorithms may also be used to eliminate the feature point pairs with matching errors from the plurality of feature point pairs and solve the homography matrix.

[0079] In one of the embodiments, the calculation formula for determining all corresponding pixel pairs in the left view and the right view based on the homography matrix is ​​specifically as follows:

[0080]

[0081] Where H is the homography matrix, is the coordinate of the pixel point in the left view, is the coordinate of the pixel point in the right view corresponding to the left view.

[0082] In one embodiment, the determining the current working distance of the binocular camera device based on the coordinates of all corresponding pixel pairs includes: calculating the average disparity d of the pixel pairs based on the coordinates of all corresponding pixel pairs; specifically, performing disparity calculation on each pixel pair to obtain the corresponding disparity value, and then calculating the average disparity d of all pixel pairs in the region of interest. Then, the depth information of the object being photographed in the current region of interest is obtained using the following formula, that is, the current working distance z of the binocular camera device:

[0083]

[0084] Where f is the focal length of the camera, b is the baseline distance, and d is the average disparity of all pixel pairs. If the maximum and minimum disparity of all pixel pairs in the region of interest are used to calculate the working distance, the range of the current working distance can be obtained.

[0085] This application counts the depth values ​​of all pixels in the area of ​​interest, and can generate a depth map with almost no holes, thereby improving the accuracy of calculating the current camera working distance.

[0086] In one embodiment, the step of obtaining the displacement adjustment value of the left view and the displacement adjustment value of the right view based on the current working distance of the binocular camera device and the image displacement table comprises: querying the image displacement table based on the current working distance of the binocular camera device to obtain a displacement query result; and interpolating the displacement query result to obtain the displacement adjustment value of the left view and the displacement adjustment value of the right view. Specifically, the steps are as follows:

[0087] Step S501, based on the current working distance of the binocular shooting device, determining a first working distance and a second working distance adjacent to the current working distance in the image displacement table;

[0088] Step S502, querying the image displacement table to determine a first displacement adjustment value corresponding to the first working distance and a second displacement adjustment value corresponding to the second working distance;

[0089] Specifically, the first displacement adjustment value includes the displacement adjustment value corresponding to the left view and the displacement adjustment value corresponding to the right view at the first working distance; the second displacement adjustment value includes the displacement adjustment value corresponding to the left view and the displacement adjustment value corresponding to the right view at the second working distance.

[0090] Step S503: performing interpolation calculation on the first displacement adjustment value and the second displacement adjustment value to obtain the displacement adjustment value of the left view and the displacement adjustment value of the right view.

[0091] Specifically, interpolation calculation is performed on the displacement adjustment value corresponding to the left view at the first working distance and the displacement adjustment value corresponding to the left view at the second working distance, respectively, to obtain the displacement adjustment value of the left view at the current working distance; interpolation calculation is performed on the displacement adjustment value corresponding to the right view at the first working distance and the displacement adjustment value corresponding to the right view at the second working distance, to obtain the displacement adjustment value of the right view at the current working distance; the displacement adjustment value is the translation value of the left and right views, and the left view and the right view are translated according to the displacement adjustment value, and parallax correction can be achieved after translation. In this implementation, the interpolation method may adopt nearest neighbor interpolation, bilinear interpolation, etc. If the current working distance of the binocular shooting device is exactly in the image displacement table, the corresponding displacement adjustment value of the left view and the displacement adjustment value of the right view can be directly obtained in the image displacement table.

[0092] In a preferred embodiment, the parallax automatic correction method of this embodiment includes the following steps:

[0093] Step S51, respectively performing displacement adjustment on the left view and the right view acquired by the binocular shooting device at different working distances until the target stereoscopic visual effect is achieved, so as to obtain an image displacement table;

[0094] Step S52, using a feature point detection algorithm and a feature point matching algorithm to obtain multiple feature point pairs in the left view and the right view at the current working distance;

[0095] Step S53, using a random sampling consensus algorithm to eliminate feature point pairs with matching errors from the multiple feature point pairs; then solving a homography matrix based on the coordinates of the multiple feature point pairs after eliminating the matching errors; and determining all corresponding pixel point pairs in the left view and the right view according to the solved homography matrix;

[0096] Step S54, calculating the disparity value of each pixel in the left view, and using the disparity formula to obtain the working distance of the current binocular shooting device;

[0097] Step S55, interpolating the table lookup result of the current working distance to obtain the displacement adjustment values ​​of the left and right views to achieve a more comfortable stereoscopic visual effect.

[0098] This application adopts the method of feature point detection, feature point matching combined with homography matrix solution to obtain the depth map of the area of ​​interest, reduce the impact of different filter window sizes on the effect and performance, have better stability, reduce computing cost consumption, improve stability and the accuracy of the obtained depth information;

[0099] This application introduces a homography matrix, which can not only avoid the situation where there are fewer depth samples due to fewer feature points, but also filter the successfully matched feature point pairs and eliminate the incorrectly matched feature point pairs;

[0100] This application counts the depth values ​​of all pixels in the area of ​​interest, and can generate a depth map with almost no holes, thereby improving the accuracy of calculating the working distance of the current binocular camera device.

[0101] The present application also provides a parallax automatic correction device, such as Figure 6 As shown, the device comprises:

[0102] A first obtaining module 610 is used to perform displacement adjustment on the left view and the right view acquired by the binocular shooting device at different working distances, respectively, until a target stereoscopic visual effect is achieved, so as to obtain an image displacement table;

[0103] A second obtaining module 620 is used to obtain multiple feature point pairs in the left view and the right view at the current working distance;

[0104] A determination module 630, configured to determine all corresponding pixel point pairs in the left view and the right view based on the coordinate mapping relationship between the multiple feature point pairs;

[0105] A third obtaining module 640 is used to determine the current working distance of the binocular shooting device based on the coordinates of all corresponding pixel pairs;

[0106] The adjustment module 650 is used to query the image displacement table based on the current working distance of the binocular shooting device, obtain the query result, and interpolate the query result to obtain the displacement adjustment value of the left view and the displacement adjustment value of the right view, so as to move and adjust the left view and the right view respectively.

[0107] In one embodiment, the second acquisition module 620 is also used to select a first region of interest in the left view and a second region of interest in the right view; use a feature point detection algorithm to perform feature point detection on the first region of interest and the second region of interest respectively to obtain corresponding feature points; use a feature point matching algorithm to match the feature points of the first region of interest with the feature points of the second region of interest to obtain multiple feature point pairs in the left view and the right view.

[0108] In one of the embodiments, the determination module 630 is further configured to solve a homography matrix based on the coordinates of the plurality of feature point pairs; and determine all corresponding pixel point pairs in the left view and the right view based on the homography matrix.

[0109] In one embodiment, the determination module 630 is further configured to remove feature point pairs with matching errors from the plurality of feature point pairs; and solve the homography matrix based on the coordinates of the plurality of feature point pairs after the matching errors are removed.

[0110] In one of the embodiments, the calculation formula for determining all corresponding pixel pairs in the left view and the right view based on the homography matrix is ​​specifically as follows:

[0111]

[0112] Where H is the homography matrix, is the coordinate of the pixel point in the left view, is the coordinate of the pixel point in the right view corresponding to the left view.

[0113] In one of the embodiments, the adjustment module 650 is also used to query the image displacement table based on the current working distance of the binocular shooting device to obtain a displacement query result; interpolate the displacement query result to obtain a displacement adjustment value of the left view and a displacement adjustment value of the right view.

[0114] In one embodiment, the adjustment module 650 is also used to: determine a first working distance and a second working distance adjacent to the current working distance in the image displacement table based on the current working distance of the binocular shooting device; query the image displacement table to determine a first displacement adjustment value corresponding to the first working distance and a second displacement adjustment value corresponding to the second working distance; and perform interpolation calculation on the first displacement adjustment value and the second displacement adjustment value to obtain a displacement adjustment value of the left view and a displacement adjustment value of the right view.

[0115] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0116] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a parallax automatic correction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0117] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any of the above-mentioned automatic parallax correction method embodiments are implemented.

[0119] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0120] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A parallax automatic correction method, It is characterized in that The method comprises: The left view and the right view obtained by the binocular shooting device at different working distances are respectively adjusted in displacement until the target stereoscopic vision effect is achieved, so as to obtain an image displacement table; Get multiple feature point pairs in the left view and the right view at the current working distance; Based on the coordinate mapping relationship between the multiple feature point pairs, determine all corresponding pixel point pairs in the left view and the right view; Determining the current working distance of the binocular camera device based on the coordinates of all corresponding pixel pairs; Based on the current working distance of the binocular shooting device and the image displacement table, the displacement adjustment value of the left view and the displacement adjustment value of the right view are determined to move and adjust the left view and the right view respectively.

2. The method according to claim 1, It is characterized in that The obtaining of multiple feature point pairs in the left view and the right view at the current working distance comprises: Selecting a first region of interest in the left view, and selecting a second region of interest in the right view; Using a feature point detection algorithm to perform feature point detection on the first region of interest and the second region of interest respectively to obtain corresponding feature points; The feature points of the first region of interest are matched with the feature points of the second region of interest using a feature point matching algorithm to obtain a plurality of feature point pairs in the left view and the right view.

3. The method according to claim 1, It is characterized in that The determining, based on the coordinate mapping relationship between the coordinates of the plurality of feature point pairs, all corresponding pixel point pairs in the left view and the right view comprises: Solving a homography matrix based on the coordinates of the plurality of feature point pairs; Based on the homography matrix, all corresponding pixel pairs in the left view and the right view are determined.

4. The method according to claim 3, It is characterized in that Solving the homography matrix based on the coordinates of the plurality of feature point pairs comprises: Eliminate feature point pairs with incorrect matching among the multiple feature point pairs; The homography matrix is ​​solved based on the coordinates of multiple feature point pairs after eliminating matching errors.

5. The method according to claim 3, It is characterized in that The calculation formula for determining all corresponding pixel pairs in the left view and the right view based on the homography matrix is ​​specifically as follows: Where H is the homography matrix, is the coordinate of the pixel point in the left view, is the coordinate of the pixel point in the right view corresponding to the left view.

6. The method according to claim 1, It is characterized in that The obtaining the displacement adjustment value of the left view and the displacement adjustment value of the right view based on the current working distance of the binocular shooting device and the image displacement table comprises: Querying the image displacement table based on the current working distance of the binocular shooting device to obtain a displacement query result; The displacement query result is interpolated to obtain a displacement adjustment value of the left view and a displacement adjustment value of the right view.

7. The method according to claim 6, It is characterized in that querying the image displacement table based on the current working distance of the binocular shooting device to obtain a displacement query result; Interpolating the displacement query result to obtain the displacement adjustment value of the left view and the displacement adjustment value of the right view includes: Based on the current working distance of the binocular shooting device, determining in the image displacement table a first working distance and a second working distance adjacent to the current working distance; Querying the image displacement table to determine a first displacement adjustment value corresponding to the first working distance and a second displacement adjustment value corresponding to the second working distance; Interpolation calculation is performed on the first displacement adjustment value and the second displacement adjustment value to obtain the displacement adjustment value of the left view and the displacement adjustment value of the right view.

8. A parallax automatic correction device, It is characterized in that The device comprises: The first acquisition module is used to respectively adjust the displacement of the left view and the right view obtained by the binocular shooting device at different working distances until the target stereoscopic vision effect is achieved, so as to obtain an image displacement table; The second acquisition module is used to obtain multiple feature point pairs in the left view and the right view at the current working distance; A determination module, configured to determine all corresponding pixel point pairs in the left view and the right view based on the coordinate mapping relationship between the plurality of feature point pairs; A third obtaining module is used to determine the current working distance of the binocular shooting device based on the coordinates of all corresponding pixel pairs; An adjustment module is used to determine the displacement adjustment value of the left view and the displacement adjustment value of the right view based on the current working distance of the binocular shooting device and the image displacement table, so as to move and adjust the left view and the right view respectively.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.