Phase Difference Detection Method, Device, Equipment and Medium Based on Feature Matching

Abstract

The present invention is applicable to the field of image processing technology, and provides a phase difference detection method, device, electronic device and storage medium based on feature matching. The method includes: obtaining original PD data pairs of each PD pixel row from an image sensor, respectively performing block processing on the left PD data and the right PD data in each group of original PD data pairs to obtain multiple groups of first PD data block pairs, using a preset difference operator to perform convolution operations on each group of first PD data block pairs to obtain multiple groups of second PD data block pairs, performing feature matching on each group of second PD data block pairs to obtain the phase difference of each group of second PD data block pairs, and generating a phase difference detection result based on the phase difference of each group of second PD data block pairs, thereby realizing the detection of the phase difference based on feature matching.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a phase difference detection method, device, electronic device and storage medium based on feature matching. Background Art

[0002] When a user takes a photo through an electronic device, the electronic device can perform focusing in a phase difference auto focus (PDAF) manner to obtain a relatively clear picture. When the electronic device performs focusing, it can calculate the phase difference through PD point pairs arranged on a sensor, and convert the phase difference into the moving distance of a motor in a lens module, so as to determine a focus point according to the moving distance to achieve focusing.

[0003] However, in the prior art, focusing is usually performed according to the original phase difference data provided by an image sensor, and the detection method of the original phase difference data is not transparent. Summary of the Invention

[0004] The purpose of the present invention is to provide a phase difference detection method, device, electronic device and storage medium based on feature matching, aiming to provide a brand-new phase difference detection method based on feature matching.

[0005] On the one hand, the present invention provides a phase difference detection method based on feature matching, and the method includes the following steps:

[0006] Obtain the original PD data pairs of each PD pixel row from an image sensor;

[0007] Perform block processing on the left PD data and the right PD data in each group of the original PD data pairs respectively to obtain multiple groups of first PD data block pairs;

[0008] Perform convolution operation on each group of the first PD data block pairs by using a preset difference operator to obtain multiple groups of second PD data block pairs;

[0009] Perform feature matching on each group of the second PD data block pairs to obtain the phase difference of each group of the second PD data block pairs;

[0010] Generate a phase difference detection result based on the phase difference of each group of the second PD data block pairs.

[0011] Preferably, before the step of performing convolution operation on each group of the first PD data block pairs by using a preset difference operator, it further includes:

[0012] Perform denoising processing on each group of the first PD data block pairs by using a preset filter to obtain multiple groups of third PD data block pairs, and the filter is a Gaussian filter;

[0013] The step of performing a convolution operation on each group of the first PD data block pairs using a preset difference operator includes:

[0014] Performing a convolution operation on each group of the third PD data block pairs using the difference operator to obtain multiple groups of the second PD data block pairs.

[0015] Preferably, the radius of the difference operator is determined according to the size of the image sensor and the distribution of PD pixels in the image sensor.

[0016] Preferably, the step of performing feature matching on each group of the second PD data block pairs to obtain the phase difference of each group of the second PD data block pairs includes:

[0017] Obtaining a pair of feature point sets of the second PD data block pair;

[0018] Performing feature point matching based on the coordinates of each feature point in the pair of feature point sets, and obtaining the phase difference of the second PD data block pair based on the coordinate differences of the matched feature point pairs.

[0019] Preferably, the pair of feature point sets includes a first feature point set of the left PD data block in the second PD data block pair and a second feature point set of the right PD data block in the second PD data block pair. The step of obtaining the pair of feature point sets of the second PD data block pair includes:

[0020] Obtaining all first extreme points of the left PD data block in the second PD data block pair and all second extreme points of the right PD data block in the second PD data block pair;

[0021] Obtaining the first feature point set based on all the first extreme points and obtaining the second feature point set based on all the second extreme points.

[0022] Preferably, the step of obtaining the first feature point set based on all the first extreme points and obtaining the second feature point set based on all the second extreme points includes:

[0023] Screening first valid extreme points with absolute pixel values greater than the first extreme point threshold from all the first extreme points, and screening second valid extreme points with absolute pixel values greater than the second extreme point threshold from all the second extreme points;

[0024] Combining all the first valid extreme points to obtain the first feature point set, and combining all the second valid extreme points to obtain the second feature point set.

[0025] Preferably, the first extreme point threshold is the product of the maximum absolute pixel value of all the first extreme points and a preset multiple, and the second extreme point threshold is the product of the maximum absolute pixel value of all the second extreme points and the preset multiple.

[0026] Preferably, before the step of performing feature point matching based on the coordinates of each feature point in the feature point set pair, it includes:

[0027] Fitting the coordinates of each feature point in the feature point set pair to obtain a feature point coordinate set pair;

[0028] The step of performing feature point matching based on the coordinates of each feature point in the feature point set pair includes:

[0029] Performing feature point matching according to the fitted coordinates of each feature point in the feature point coordinate set pair.

[0030] Preferably, the step of fitting the coordinates of each feature point in the feature point set pair includes:

[0031] Searching for a first PD pixel point with the smallest absolute value of the difference in pixel values from the feature point within the left and right neighborhoods of the feature point;

[0032] Centering on the feature point, searching for a second PD pixel point within the symmetric neighborhood of the first PD pixel point, where the absolute pixel value of the first PD pixel point is between the absolute pixel values of the left and right adjacent PD pixel points of the second PD pixel point;

[0033] Performing linear fitting on the second PD pixel point and a third PD pixel point to obtain a fitted line segment, where the third PD pixel point is the PD pixel point with the smallest absolute value of the difference in pixel values from the first PD pixel point among the left and right adjacent PD pixel points of the second PD pixel point;

[0034] Searching for a target point with the same pixel value as the first PD pixel point on the fitted line segment;

[0035] Obtaining the symmetric points of the first PD pixel point and the target point, and taking the coordinates of the symmetric points as the fitted coordinates of the feature point.

[0036] On the other hand, the present invention provides a phase difference detection device based on feature matching, and the device includes:

[0037] An original data acquisition unit, configured to acquire original PD data pairs of each PD pixel row from an image sensor;

[0038] A block processing unit, configured to perform block processing on the left PD data and the right PD data in each group of the original PD data pairs respectively to obtain multiple groups of first PD data block pairs;

[0039] A differential operation unit is configured to perform a convolution operation on each group of the first PD data block pairs by using a preset differential operator to obtain multiple groups of second PD data block pairs;

[0040] A feature matching unit is configured to perform feature matching on each group of the second PD data block pairs to obtain the phase difference of each group of the second PD data block pairs; and

[0041] A detection result generation unit is configured to generate a phase difference detection result based on the phase difference of each group of the second PD data block pairs.

[0042] On the other hand, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0043] On the other hand, the present invention further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0044] The present invention obtains the original PD data pairs of each PD pixel row from an image sensor, respectively performs block processing on the left PD data and the right PD data in each group of original PD data pairs to obtain multiple groups of first PD data block pairs, performs a convolution operation on each group of first PD data block pairs by using a preset differential operator to obtain multiple groups of second PD data block pairs, performs feature matching on each group of second PD data block pairs to obtain the phase difference of each group of second PD data block pairs, and generates a phase difference detection result based on the phase difference of each group of second PD data block pairs, thereby realizing the detection of the phase difference based on feature matching. Description of the Drawings

[0045] Figure 1A is a flowchart of the implementation of the phase difference detection method based on feature matching provided in the first embodiment of the present invention;

[0046] Figure 1B is a layout diagram of PD pixels of the IMX455 sensor of sony provided in the first embodiment of the present invention;

[0047] Figure 1C is an example diagram of pixel information of the first PD data block pair provided in the first embodiment of the present invention;

[0048] Figure 1D is an example diagram of pixel information of a PD data block before and after filtering provided in the first embodiment of the present invention;

[0049] Figure 1EIt is an example diagram of the pixel information of the second PD data block pair obtained after differential operation provided in the first embodiment of the present invention;

[0050] Figure 1F It is an example diagram of the coordinate difference between the matching feature point pairs provided in the first embodiment of the present invention;

[0051] Figure 1G It is an example diagram of the extreme points of the left and right PD data blocks in the second PD data block pair provided in the first embodiment of the present invention;

[0052] Figure 1H It is an example diagram of the valid extreme points of the left and right PD data blocks in the second PD data block pair provided in the first embodiment of the present invention;

[0053] Figure 1I It is an example diagram of the coordinate of the feature points before and after fitting provided in the first embodiment of the present invention;

[0054] Figure 2 It is a schematic structural diagram of a phase difference detection device based on feature matching provided in the second embodiment of the present invention; and

[0055] Figure 3 It is a schematic structural diagram of an electronic device provided in the third embodiment of the present invention. Detailed implementation manners

[0056] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] The following describes the specific implementation of the present invention in detail in combination with specific embodiments:

[0058] Example 1:

[0059] Figure 1A The implementation process of the phase difference detection method based on feature matching provided in the first embodiment of the present invention is shown. For the convenience of description, only the parts related to the embodiments of the present invention are shown and are described in detail as follows:

[0060] In step S101, the original PD data pairs of each PD pixel row are obtained from the image sensor.

[0061] The embodiments of the present invention are applicable to electronic devices, which can be terminal devices such as mobile phones, cameras, tablet computers, wearable devices, vehicle-mounted devices, laptop computers, etc. The specific types of electronic devices in the embodiments of the present application are not limited in any way.

[0062] In an embodiment of the present invention, after obtaining pixel data from an image sensor, PD pixel rows can be extracted to obtain the original PD data of each PD pixel row. The original PD data of each PD pixel row is separated into left and right PD data to obtain an original PD data pair for each PD pixel row, that is, the original PD data pair includes the original left PD data and right PD data.

[0063] In step S102, the left PD data and the right PD data in each group of original PD data pairs are respectively block-processed to obtain multiple groups of first PD data block pairs.

[0064] In an embodiment of the present invention, the left PD data and the right PD data of each PD pixel row are respectively divided into N blocks to obtain N left PD data blocks and N right PD data blocks. For the convenience of description, in this embodiment, each left PD data block obtained after block-processing is described by a first left PD data block, and each right PD data block obtained after block-processing is described by a first right PD data block. The PD data block pair composed of the first left PD data block and the corresponding first right PD data block is the above-mentioned first PD data block pair.

[0065] Taking the sony IMX455 sensor as an example, the layout of the PD pixels of the sony IMX455 sensor is as Figure 1B shown. There are 31 rows of PD pixels in this image sensor, which are evenly distributed on the imaging surface. After block-processing, 31*N first left PD pixel blocks and 31*N first right PD pixel blocks can be obtained. The pixel information of the first PD data block pair obtained after block-processing is as Figure 1C shown, where the abscissa represents the coordinates of the PD pixels, and the ordinate represents the pixel values of the PD pixels.

[0066] In step S103, a preset difference operator is used to perform a convolution operation on each group of first PD data block pairs to obtain multiple groups of second PD data block pairs.

[0067] In an embodiment of the present invention, a preset difference operator is used to perform convolution operations on the first left PD data block and the first right PD data block in each group of first PD data block pairs. For ease of description, the data blocks obtained after performing convolution operations on the first left PD data block and the first right PD data block are described as the second left PD data block and the second right PD data block respectively. The PD data block pair composed of the second left PD data block and the corresponding second right PD data block is the above-mentioned second PD data block pair. Specifically, a difference operator with a radius of 1 and a step size of 1 can be used to perform convolution operations on each group of first PD data block pairs. Preferably, the radius of the difference operator is determined according to the size of the image sensor and the distribution of PD pixels in the image sensor to improve the accuracy of subsequent phase difference calculation. Taking the sony IMX455 sensor as an example, the experimental results show that the optimal radius of the difference operator is 5 and the step size is 1, that is, the difference operator is specifically [-1, -1, -1, -1, -1, 0, 1, 1, 1, 1, 1].

[0068] Considering the imperfections of the imaging system, transmission medium, recording device, etc., digital images are often contaminated by various noises during their formation, transmission, and recording processes. Before using the preset difference operator to perform convolution operations on each group of first PD data block pairs, preferably, a preset filter is used to perform denoising processing on each group of first PD data block pairs to obtain multiple groups of third PD data block pairs to remove noise interference. Specifically, a preset filter can be used to perform denoising processing on the first left PD data block and the first right PD data block in each group of first PD data block pairs respectively. For ease of description, the PD data blocks after denoising processing are described as the third left PD data block and the third right PD data block respectively. The PD data block pair composed of the third left PD data block and the corresponding third right PD data block is the above-mentioned third PD data block pair. Figure 1D Fig. is an example diagram of pixel information before and after filtering a PD data block, where the abscissa represents the coordinates of PD pixels and the ordinate represents the pixel values of PD pixels. After obtaining multiple groups of third PD data block pairs, a difference operator is used to perform convolution operations on each group of third PD data block pairs to obtain multiple groups of second PD data block pairs. After using the difference operator to perform convolution operations on the third PD data block pairs, the pixel information of the obtained second PD data block pairs is as Figure 1E shown, where the abscissa represents the coordinates of PD pixels and the ordinate represents the pixel values of PD pixels.

[0069] Among them, the above-mentioned filter can be a mean filter or a median filter, etc. Preferably, the filter is a Gaussian filter to ensure that while not changing the edge direction of the original PD data block, it also ensures the characteristics of subsequent feature points and edges, thereby improving the accuracy of subsequent phase difference detection.

[0070] In step S104, feature matching is performed on each group of second PD data block pairs to obtain the phase difference of each group of second PD data block pairs.

[0071] In the embodiment of the present invention, when performing feature matching on each group of second PD data block pairs, preferably, a set of feature point pairs of the second PD data block pairs is obtained, feature point matching is performed based on the coordinates of each feature point in the set of feature point pairs, and the phase difference of the second PD data block pairs is obtained based on the coordinate differences of the matching feature point pairs, so that feature matching is realized according to the coordinates of the obtained feature points. Among them, the set of feature point pairs includes the first set of feature points of the left PD data block in the second PD data block pair and the second set of feature points of the right PD data block in the second PD data block pair.

[0072] Specifically, after obtaining the set of feature point pairs, for any feature point (the first feature point) in the first set of feature points, a feature point (the second feature point) with the smallest absolute value of the coordinate difference from this feature point can be searched in the corresponding second set of feature points, and the found feature point (the second feature point) is used as the matching feature point of the above feature point (the first feature point). After obtaining the matching feature point pairs, as Figure 1F shown, the phase difference of the second PD data block pair is obtained based on the coordinate difference PD between the matching feature point pairs.

[0073] It should be noted here that when both the above first set of feature points and the second set of feature points contain only one feature point, the coordinate difference of this matching feature point pair is used as the phase difference of this second PD data block pair. When both the above first set of feature points and the second set of feature points contain multiple feature points, the average value of the coordinate differences of all matching feature point pairs can be calculated, and the obtained average value is used as the phase difference of this second PD data block pair to improve the accuracy of phase difference calculation.

[0074] When obtaining the set of feature point pairs of the second PD data block pairs, preferably, all first extreme points of the left PD data block (the second left PD data block) in the second PD data block pair and all second extreme points of the right PD data block (the second right PD data block) in the second PD data block pair are obtained. The first set of feature points is obtained based on all the first extreme points, and the second set of feature points is obtained based on all the second extreme points, so that the acquisition of feature points is realized through extreme points, and the complexity of subsequent phase difference calculation is simplified. Specifically, all the first extreme points can be combined to obtain the first set of feature points, and all the second extreme points can be combined to obtain the second set of feature points. It should be pointed out here that the first and second extreme points in this embodiment are only used to distinguish the extreme points of the left and right PD data blocks in the second PD data block pair. As an example, as Figure 1G shown, Figure 1G the first extreme point of the left PD data block in the second PD data block pair in [figure number] is A, and the second extreme point of the right PD data is B.

[0075] Considering that in a complex environment, multiple extreme points will be generated, and some of the extreme points are noise or low-contrast edges. When obtaining the first set of feature points based on all the first extreme points and the second set of feature points based on all the second extreme points, preferably, first effective extreme points with absolute pixel values greater than the first extreme point threshold are selected from all the first extreme points, and second effective extreme points with absolute pixel values greater than the second extreme point threshold are selected from all the second extreme points. All the first effective extreme points are combined to obtain the first set of feature points, and all the second effective extreme points are combined to obtain the second set of feature points, thereby improving the accuracy of feature point acquisition and further improving the accuracy of subsequent phase difference calculation.

[0076] Among them, the above-mentioned first extreme point threshold can be determined according to the absolute pixel values of all the first extreme points, and the second extreme point threshold can be determined according to the absolute pixel values of all the second extreme points. Preferably, the first extreme point threshold is the product of the maximum absolute pixel value of all the first extreme points and a preset multiple, and the second extreme point threshold is the product of the maximum absolute pixel value of all the second extreme points and a preset multiple, so as to determine the extreme point threshold according to the maximum absolute pixel value of all the extreme points, thereby improving the accuracy of extreme point threshold setting. Further, the above-mentioned preset multiple is 0.5, so as to determine the above-mentioned preset multiple according to the actual experimental results and further improve the accuracy of feature point acquisition.

[0077] As an example, as Figure 1H shown, if in the left and right PD data blocks, the extreme points with the maximum absolute pixel values are A(7, -319.3) and B(10, -405.7) respectively, then the first extreme point threshold and the second extreme point threshold are |-319.3| / 2 = 159.65 and |-405.7| / 2 = 202.85 respectively. The first effective extreme points in the left PD data block that meet the above first value point threshold are A and C, and the second effective extreme points in the right PD data block that meet the above second value point threshold are B and D.

[0078] Before performing feature point matching on the coordinates of each feature point in the pair of feature point sets, preferably, the coordinates of each feature point in the pair of feature point sets are fitted to obtain a pair of feature point coordinate sets, so as to obtain feature point coordinates with higher accuracy. After obtaining the pair of feature point coordinate sets, feature point matching is performed according to the fitted coordinates of each feature point in the pair of feature point coordinate sets.

[0079] When fitting the coordinates of each feature point in the set of paired feature points, preferably, the first PD pixel point with the smallest absolute value of the difference in pixel values from the feature point is searched for within the left and right neighborhoods of the feature point. With the feature point as the center, the second PD pixel point is searched for within the symmetric neighborhood of the first PD pixel point. The second PD pixel point and the third PD pixel point are linearly fitted to obtain a fitted line segment. On the fitted line segment, the target point with the same pixel value as the first PD pixel point is searched for, the symmetric points of the first PD pixel point and the target point are obtained, and the coordinates of the symmetric points are used as the fitted coordinates of the feature point, thereby improving the accuracy of the fitted coordinates of the feature point. Among them, the absolute pixel value of the first PD pixel point is between the absolute pixel values of the left and right adjacent PD pixel points of the second PD pixel point, and the third PD pixel point is the PD pixel point with the smallest absolute value of the difference in pixel values from the first PD pixel point among the left and right adjacent PD pixel points of the second PD pixel point, that is, the PD pixel point closest to the feature point among the left and right adjacent PD pixel points of the second PD pixel point.

[0080] As an example, as Figure 1I shown, Figure 1I in which the extreme point is k, the point with the smallest absolute value of the difference from the extreme point is set as n, and n is the above-mentioned first PD pixel point. With k as the symmetric center, data is traversed in the symmetric domain, and the point m satisfying |[m - 1]| < |[n]| < |[m + 1]| is found, and m is the above-mentioned second PD pixel point. By linearly fitting the point m - 1 (i.e., the above-mentioned third PD pixel point) and the point m, the point n2 on the fitted line segment is calculated, [n2] = [n], and n2 is the above-mentioned target point. The coordinates of the symmetric point peak of n and n2 are the fitted coordinates of the feature point. Among them, [] represents the PD pixel value, and || represents the absolute value.

[0081] In step S105, a phase difference detection result is generated based on the phase difference of each pair of second PD data blocks.

[0082] In the embodiment of the present invention, the generated phase difference detection result is a matrix, and the elements of the matrix are the phase differences of each pair of second PD data blocks. Among them, the number of rows of the matrix is equal to the number of PD pixel rows, and the number of columns of the matrix is the number of blocks of the left PD data block or the right PD data block of each PD pixel row after block processing. Taking the above-mentioned sony's IMX455 sensor as an example, if 31 * N first left PD pixel blocks and 31 * N first right PD pixel blocks are obtained after block processing, the final phase difference detection result is a matrix composed of 31 * N phase differences.

[0083] Further, after obtaining the above-mentioned phase difference detection result, focusing can be performed based on the above-mentioned phase difference detection result, or depth detection can be performed based on the above-mentioned phase difference detection result.

[0084] In an embodiment of the present invention, the original PD data pairs of each PD pixel row are obtained from an image sensor, the left PD data and the right PD data in each group of original PD data pairs are respectively subjected to block processing to obtain multiple groups of first PD data block pairs, a preset difference operator is used to perform convolution operation on each group of first PD data block pairs to obtain multiple groups of second PD data block pairs, feature matching is performed on each group of second PD data block pairs to obtain the phase difference of each group of second PD data block pairs, and a phase difference detection result is generated based on the phase difference of each group of second PD data block pairs, so that the detection of the phase difference is realized based on feature matching.

[0085] Example 2:

[0086] Figure 2 FIG. shows the structure of the phase difference detection device based on feature matching provided in the second embodiment of the present invention. For ease of description, only the parts related to the embodiment of the present invention are shown, including:

[0087] An original data acquisition unit 21, configured to obtain the original PD data pairs of each PD pixel row from an image sensor;

[0088] A block processing unit 22, configured to respectively perform block processing on the left PD data and the right PD data in each group of original PD data pairs to obtain multiple groups of first PD data block pairs;

[0089] A difference operation unit 23, configured to perform convolution operation on each group of first PD data block pairs by using a preset difference operator to obtain multiple groups of second PD data block pairs;

[0090] A feature matching unit 24, configured to perform feature matching on each group of second PD data block pairs to obtain the phase difference of each group of second PD data block pairs; and

[0091] A detection result generation unit 25, configured to generate a phase difference detection result based on the phase difference of each group of second PD data block pairs.

[0092] Preferably, the device further includes:

[0093] A denoising processing unit, configured to perform denoising processing on each group of first PD data block pairs by using a preset filter to obtain multiple groups of third PD data block pairs, and the filter is a Gaussian filter;

[0094] The difference operation unit further includes;

[0095] A difference operation sub-unit, configured to perform convolution operation on each group of third PD data block pairs by using a difference operator to obtain multiple groups of second PD data block pairs.

[0096] Preferably, the radius of the difference operator is determined according to the size of the image sensor and the distribution of PD pixels in the image sensor.

[0097] Preferably, the feature matching unit includes:

[0098] A feature point acquisition unit for acquiring a pair of feature point sets of the second PD data block pair;

[0099] A feature point matching subunit for performing feature point matching based on the coordinates of each feature point in the pair of feature point sets, and obtaining the phase difference of the second PD data block pair based on the coordinate differences of the matched feature point pairs.

[0100] Preferably, the pair of feature point sets includes a first feature point set of the left PD data block in the second PD data block pair and a second feature point set of the right PD data block in the second PD data block pair. The feature point acquisition unit includes:

[0101] An extreme point acquisition unit for acquiring all first extreme points of the left PD data block in the second PD data block pair and all second extreme points of the right PD data block in the second PD data block pair; and

[0102] A feature point set generation unit for obtaining a first feature point set based on all the first extreme points and obtaining a second feature point set based on all the second extreme points.

[0103] Preferably, the feature point set generation unit further includes:

[0104] An extreme point screening unit for screening first valid extreme points with absolute pixel values greater than the first extreme point threshold from all the first extreme points and screening second valid extreme points with absolute pixel values greater than the second extreme point threshold from all the second extreme points;

[0105] A feature point set generation subunit for combining all the first valid extreme points to obtain a first feature point set and combining all the second valid extreme points to obtain a second feature point set.

[0106] Preferably, the first extreme point threshold is the product of the maximum absolute pixel value of all the first extreme points and a preset multiple, and the second extreme point threshold is the product of the maximum absolute pixel value of all the second extreme points and a preset multiple.

[0107] Preferably, the feature point matching unit further includes:

[0108] A coordinate fitting unit for fitting the coordinates of each feature point in the pair of feature point sets to obtain a pair of feature point coordinate sets;

[0109] The feature point matching unit is further configured to perform feature point matching according to the fitted coordinates of each feature point in the pair of feature point coordinate sets.

[0110] Preferably, the coordinate fitting unit includes:

[0111] The first PD point searching unit is configured to search for a first PD pixel point with the smallest absolute value of the difference from the pixel value of the feature point within the left and right neighborhoods of the feature point;

[0112] The second PD point searching unit is configured to search for a second PD pixel point within the symmetric neighborhood of the first PD pixel point centered on the feature point, where the absolute pixel value of the first PD pixel point is between the absolute pixel values of the left and right adjacent PD pixel points of the second PD pixel point;

[0113] The linear fitting unit is configured to perform linear fitting on the second PD pixel point and the third PD pixel point to obtain a fitting line segment, where the third PD pixel point is the PD pixel point with the smallest absolute value of the difference from the pixel value of the first PD pixel point among the left and right adjacent PD pixel points of the second PD pixel point;

[0114] The target point searching unit is configured to search for a target point with the same pixel value as the first PD pixel point on the fitting line segment; and

[0115] The symmetric point obtaining unit is configured to obtain the symmetric points of the first PD pixel point and the target point, and use the coordinates of the symmetric points as the fitting coordinates of the feature point.

[0116] In the embodiments of the present invention, each unit of the phase difference detection device based on feature matching can be implemented by corresponding hardware or software units. Each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which is not intended to limit the present invention here. The specific implementation manners of each unit of the phase difference detection device based on feature matching can refer to the description of the foregoing method embodiments, and will not be elaborated here.

[0117] Example 3:

[0118] Figure 3 The structure of the electronic device provided in Embodiment 3 of the present invention is shown. For the sake of convenience of description, only the parts related to the embodiments of the present invention are shown.

[0119] The electronic device 3 in the embodiments of the present invention includes a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, the steps in the foregoing method embodiments are implemented. For example, Figure 1A The steps S101 to S105 shown. Alternatively, when the processor 30 executes the computer program 32, the functions of each unit in the foregoing device embodiments are implemented. For example, Figure 2 The functions of the units 21 to 25 shown.

[0120] In an embodiment of the present invention, the original PD data pairs of each PD pixel row are obtained from an image sensor, the left PD data and the right PD data in each group of original PD data pairs are respectively block - processed to obtain multiple groups of first PD data block pairs, a preset difference operator is used to perform a convolution operation on each group of first PD data block pairs to obtain multiple groups of second PD data block pairs, feature matching is performed on each group of second PD data block pairs to obtain the phase difference of each group of second PD data block pairs, and a phase - difference detection result is generated based on the phase difference of each group of second PD data block pairs, thereby realizing the detection of the phase difference based on feature matching.

[0121] Example 4:

[0122] In an embodiment of the present invention, a computer - readable storage medium is provided. The computer - readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above - mentioned method embodiments are implemented. For example, Figure 1A the steps S101 to S105 shown. Or, when the computer program is executed by a processor, the functions of each unit in the above - mentioned device embodiments are implemented. For example, Figure 2 the functions of the units 21 to 25 shown.

[0123] In an embodiment of the present invention, the original PD data pairs of each PD pixel row are obtained from an image sensor, the left PD data and the right PD data in each group of original PD data pairs are respectively block - processed to obtain multiple groups of first PD data block pairs, a preset difference operator is used to perform a convolution operation on each group of first PD data block pairs to obtain multiple groups of second PD data block pairs, feature matching is performed on each group of second PD data block pairs to obtain the phase difference of each group of second PD data block pairs, and a phase - difference detection result is generated based on the phase difference of each group of second PD data block pairs, thereby realizing the detection of the phase difference based on feature matching.

[0124] The computer - readable storage medium of the embodiment of the present invention may include any entity or device, recording medium that can carry computer program code, such as memories like ROM / RAM, disks, optical discs, flash memories, etc.

[0125] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A phase difference detection method based on feature matching, characterized in that The method includes the following steps: Obtain the original PD data pairs of each PD pixel row from the image sensor; Perform block processing on the left PD data and the right PD data in each group of the original PD data pairs respectively to obtain multiple groups of first PD data block pairs; Perform convolution operation on each group of the first PD data block pairs using a preset difference operator to obtain multiple groups of second PD data block pairs, where the difference operator includes a radius and a step size for convolution operation; Perform feature matching on each group of the second PD data block pairs to obtain the phase difference of each group of the second PD data block pairs; Generate a phase difference detection result based on the phase difference of each group of the second PD data block pairs; Among them, the step of performing feature matching on each group of the second PD data block pairs to obtain the phase difference of each group of the second PD data block pairs includes: Obtain a pair of feature point sets of the second PD data block pair, where the pair of feature point sets includes a first feature point set of the left PD data block in the second PD data block pair and a second feature point set of the right PD data block in the second PD data block pair; Perform feature point matching based on the coordinates of each feature point in the pair of feature point sets, and obtain the phase difference of the second PD data block pair based on the coordinate difference of the matching feature point pairs; The step of obtaining the pair of feature point sets of the second PD data block pair includes: Obtain all first extreme points of the left PD data block in the second PD data block pair and all second extreme points of the right PD data block in the second PD data block pair; Obtain the first feature point set based on all the first extreme points, and obtain the second feature point set based on all the second extreme points.

2. The method according to claim 1, characterized in that Before the step of performing convolution operation on each group of the first PD data block pairs using a preset difference operator, it further includes: Perform denoising processing on each group of the first PD data block pairs using a preset filter to obtain multiple groups of third PD data block pairs, and the filter is a Gaussian filter; The step of performing convolution operation on each group of the first PD data block pairs using a preset difference operator includes; Perform convolution operation on each group of the third PD data block pairs using the difference operator to obtain multiple groups of the second PD data block pairs.

3. The method according to claim 1, wherein The radius of the difference operator is determined according to the size of the image sensor and the distribution of PD pixels in the image sensor.

4. The method according to claim 1, wherein The step of obtaining the first feature point set based on all the first extreme points and obtaining the second feature point set based on all the second extreme points includes: Screen out first valid extreme points with absolute pixel values greater than the first extreme point threshold from all the first extreme points, and screen out second valid extreme points with absolute pixel values greater than the second extreme point threshold from all the second extreme points; Combine all the first valid extreme points to obtain the first feature point set, and combine all the second valid extreme points to obtain the second feature point set.

5. The method according to claim 4, wherein The first extreme point threshold is the product of the maximum absolute pixel value of all the first extreme points and a preset multiple, and the second extreme point threshold is the product of the maximum absolute pixel value of all the second extreme points and the preset multiple.

6. The method according to claim 1, characterized in that, Before the step of performing feature point matching on the coordinates of each feature point in the feature point set pair, it includes: Fitting the coordinates of each feature point in the feature point set pair to obtain a pair of feature point coordinate sets; The step of performing feature point matching based on the coordinates of each feature point in the feature point set pair includes: Performing feature point matching according to the fitted coordinates of each feature point in the pair of feature point coordinate sets.

7. The method according to claim 6, wherein The step of fitting the coordinates of each feature point in the feature point set pair includes: Searching for the first PD pixel point with the smallest absolute value of the difference in pixel values from the feature point within the left and right neighborhoods of the feature point; Centering on the feature point, searching for a second PD pixel point within the symmetric neighborhood of the first PD pixel point, where the absolute pixel value of the first PD pixel point is between the absolute pixel values of the left and right adjacent PD pixel points of the second PD pixel point; Performing linear fitting on the second PD pixel point and the third PD pixel point to obtain a fitted line segment, where the third PD pixel point is the PD pixel point with the smallest absolute value of the difference in pixel values from the first PD pixel point among the left and right adjacent PD pixel points of the second PD pixel point; Searching for a target point with the same pixel value as the first PD pixel point on the fitted line segment; Obtaining the symmetric points of the first PD pixel point and the target point, and taking the coordinates of the symmetric points as the fitted coordinates of the feature point.

8. A phase difference detection device based on feature matching, characterized in that The device includes: An original data acquisition unit for acquiring a pair of original PD data for each PD pixel row from an image sensor; A block processing unit for respectively performing block processing on the left PD data and the right PD data in each group of the original PD data pairs to obtain multiple groups of first PD data block pairs; A differential operation unit for performing convolution operation on each group of the first PD data block pairs using a preset differential operator to obtain multiple groups of second PD data block pairs, where the differential operator includes a radius and a step size for convolution operation; A feature matching unit for performing feature matching on each group of the second PD data block pairs to obtain the phase difference of each group of the second PD data block pairs, including: obtaining all the first extreme points of the left PD data block in the second PD data block pair, and all the second extreme points of the right PD data block in the second PD data block pair; obtaining a first feature point set of the left PD data block based on all the first extreme points, and obtaining a second feature point set of the right PD data block based on all the second extreme points; performing feature point matching based on the coordinates of each feature point in the first feature point set of the left PD data block and the second feature point set of the right PD data block, and obtaining the phase difference of the second PD data block pair based on the coordinate difference of the matching feature point pairs; and A detection result generation unit for generating a phase difference detection result based on the phase difference of each group of the second PD data block pairs.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images