Focusing method, focusing device and storage medium
Through the depth estimation neural network, the left and right views in the preview screen collected by the camera are processed, and the parallax probability distribution and confidence are determined, which solves the problem of inaccurate focus in dark light or overexposure environments, and improves the stability of camera focus and accuracy.
Patent Information
- Application Number
- CN202310638056.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-05-31
AI Technical Summary
In low light or overexposure environments, during phase difference autofocus (PDAF), the calculation of parallax is difficult to accurately reflect the real and stable parallax, resulting in inaccurate camera focus.
By extracting the original images of the left and right view in the preview screen acquired by the camera, the depth estimation neural network is used to determine the parallax probability distribution and confidence, and focus adjustment is performed based on the confidence threshold.
It realizes stable and reliable parallax calculation in dark light or overexposure environments, reduces focus loss and improves the user's photography experience.
Smart Images

Figure CN119071629A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of cameras, and in particular to a focusing method, a focusing device and a storage medium. Background Art
[0002] In the related technology, the calculation of parallax is crucial in the Phase Difference Auto Focus (PDAF) process, which is directly related to whether the focus is clear. The parallax calculated by the related algorithm in low light or overexposed environments and the confidence of the parallax are difficult to accurately reflect the real and stable parallax, resulting in the camera being unable to focus accurately. Summary of the invention
[0003] In order to overcome the problems existing in the related art, the present disclosure provides a focusing method, a focusing device and a storage medium.
[0004] According to a first aspect of an embodiment of the present disclosure, a focusing method is provided, which is applied to a terminal, and the method includes:
[0005] In response to the preview picture captured by the image capture device in the terminal being triggered to focus, obtaining a left view original image and a right view original image corresponding to the focus area based on the focus area in the preview picture;
[0006] Determine a current disparity between the left view original image and the right view original image, and determine a disparity probability distribution corresponding to the current disparity, the disparity probability distribution characterizing a correspondence between a plurality of different disparities and probability values, the plurality of different disparities including the current disparity and a plurality of disparities with different disparity offsets of the current disparity, the probability value characterizing a degree of matching between the different disparities and the current disparity; determine a confidence level of the current disparity based on the disparity probability distribution; and perform focusing based on the disparity in response to the confidence level satisfying a confidence threshold.
[0007] In one embodiment, determining the current disparity between the left view original image and the right view original image includes: extracting image features of the left view original image to obtain a left view feature image, and extracting image features of the right view original image to obtain a right view feature image; determining a first disparity map between the left view feature image and the right view feature image, the first disparity map representing the disparity between feature points in the left view feature image and the right view feature image; upsampling the first disparity map to the original image size to obtain a second disparity map between the left view original image and the right view original image, the second disparity map representing the disparity between pixel points in the left view feature image and the right view feature image; and determining the current disparity between the left view and the right view based on the second disparity map.
[0008] In one embodiment, determining the first disparity map between the left view feature image and the right view feature image includes: for each feature point in the left view feature image and the right view feature image, respectively, as a target feature point, determining the first disparity of the target feature point; based on the first disparity of each target feature point, obtaining a first disparity map; wherein determining the first disparity of the target feature point includes: determining the similarity of the target feature point in the left view feature image and the right view feature image under multiple different disparities, and predicting the disparity corresponding to the similarity under the multiple different disparities as the probability value of the first disparity; taking the disparity corresponding to the highest probability value as the first disparity of the target feature point.
[0009] In one embodiment, determining the similarity of target feature points in the left view feature image and the right view feature image under multiple different parallaxes includes: taking one of the left view feature image and the right view feature image as a reference image and the other as an image to be matched; mapping the target feature points in the reference image to the image to be matched under multiple different parallaxes to obtain multiple target feature points to be matched corresponding to the multiple different parallaxes; and taking the similarity between the target feature point and the multiple target feature points to be matched as the similarity of the target feature point under multiple different parallaxes.
[0010] In one implementation, the disparity probability distribution is determined based on the first disparity and the predicted probability value of the disparity corresponding to the similarity under the plurality of different disparities as the first disparity.
[0011] In one embodiment, the extracting image features of the left view original image to obtain a left view feature image, and extracting image features of the right view original image to obtain a right view feature image, includes: inputting the left view original image and the right view original image into a first network model respectively, and performing weight sharing to extract image features of the left view original image and image features of the right view original image to obtain a left view feature image and a right view feature image; the input of the first network model is an image, and the output is an image feature.
[0012] In one embodiment, determining the first disparity of the target feature point includes: inputting the target feature point into a second network model, the input of the second network model is the feature point, and the output is the disparity; and obtaining the first disparity based on the output of the second network model.
[0013] In one embodiment, the second network model is trained in the following manner: obtaining an original image sequence; determining the clearest image from the original image sequence, obtaining a position corresponding to the clearest image, and determining a disparity map of each image corresponding to the position based on a proportional relationship between the position and the disparity; and training the second network model based on the image sequence and the corresponding disparity map.
[0014] In one embodiment, determining the confidence of the current disparity based on the disparity probability distribution includes: determining the disparity probability distribution curve based on the disparity probability distribution; obtaining a first area through the disparity probability distribution curve, with the probability value corresponding to the current disparity value as high and the difference between the maximum and minimum values of the different disparities as wide; determining a second area formed by the disparity probability distribution curve and the disparity axis between the maximum and minimum values of different disparities; and determining the confidence of the current disparity based on the first area and the second area.
[0015] According to a second aspect of an embodiment of the present disclosure, a focusing device is provided, which is applied to a terminal, and the focusing device includes:
[0016] The acquisition unit is used for obtaining a left view original image and a right view original image corresponding to the focus area based on the focus area in the preview image in response to the preview image acquired by the image acquisition device being triggered to perform focusing.
[0017] A disparity determination unit is used to determine the current disparity between the left view original image and the right view original image, and determine the disparity probability distribution corresponding to the current disparity, the disparity probability distribution represents the correspondence between multiple different disparities and probability values, the multiple different disparities include the current disparity and multiple disparities with different disparity offsets of the current disparity, and the probability value represents the degree of matching between the different disparities and the current disparity.
[0018] A confidence determination unit is used to determine the confidence of the current disparity based on the disparity probability distribution.
[0019] A focusing unit is configured to focus based on the parallax in response to the confidence level satisfying a confidence level threshold.
[0020] In one embodiment, the disparity determination unit determines the current disparity between the left view original image and the right view original image in the following manner: extracting image features of the left view original image to obtain a left view feature image, and extracting image features of the right view original image to obtain a right view feature image; determining a first disparity map between the left view feature image and the right view feature image, the first disparity map representing the disparity between feature points in the left view feature image and the right view feature image; upsampling the first disparity map to the original image size to obtain a second disparity map between the left view original image and the right view original image, the second disparity map representing the disparity between pixel points in the left view feature image and the right view feature image; and determining the current disparity between the left view and the right view based on the second disparity map.
[0021] In one embodiment, the disparity determination unit determines the first disparity map between the left view feature image and the right view feature image in the following manner: for each feature point in the left view feature image and the right view feature image, respectively, the first disparity of the target feature point is determined; based on the first disparity of each target feature point, a first disparity map is obtained; wherein, determining the first disparity of the target feature point includes: determining the similarity of the target feature point in the left view feature image and the right view feature image under multiple different disparities, and predicting the disparity corresponding to the similarity under the multiple different disparities as the probability value of the first disparity; taking the disparity corresponding to the highest probability value as the first disparity of the target feature point.
[0022] In one embodiment, the disparity determination unit determines the similarity of target feature points in the left-view feature image and the right-view feature image under multiple different disparities in the following manner: one of the left-view feature image and the right-view feature image is used as a reference image, and the other is used as a to-be-matched image; the target feature points in the reference image are mapped to the to-be-matched image under multiple different disparities to obtain multiple to-be-matched target feature points corresponding to the multiple different disparities; the similarity between the target feature point and the multiple to-be-matched target feature points is used as the similarity of the target feature point under multiple different parallaxes.
[0023] In one implementation, the disparity determination unit is further configured to determine the disparity probability distribution based on the first disparity and the predicted similarity-corresponding disparities under the plurality of different disparities as a probability value of the first disparity.
[0024] In one implementation, the disparity determination unit extracts image features of the left view original image to obtain a left view feature image, and extracts image features of the right view original image to obtain a right view feature image in the following manner:
[0025] The left view original image and the right view original image are respectively input into a first network model, and weight sharing is performed to extract image features of the left view original image and image features of the right view original image, so as to obtain a left view feature image and a right view feature image; the input of the first network model is an image, and the output is an image feature.
[0026] In one embodiment, the disparity determination unit determines the first disparity of the target feature point in the following manner: inputting the target feature point into a second network model, the input of the second network model is the feature point, and the output is the disparity; and obtaining the first disparity based on the output of the second network model.
[0027] In one embodiment, the disparity determination unit trains the second network model in the following manner: obtain an original image sequence; determine the clearest image from the original image sequence, obtain the position corresponding to the clearest image, and determine the disparity map of each image corresponding to the position based on the proportional relationship between the position and the disparity; and train the second network model based on the image sequence and the corresponding disparity map.
[0028] In one embodiment, the confidence determination unit determines the confidence of the current disparity based on the disparity probability distribution in the following manner: based on the disparity probability distribution, determine the disparity probability distribution curve; through the disparity probability distribution curve, obtain a first area with the probability value corresponding to the current disparity value as high and the difference between the maximum and minimum values of the different disparities as wide; determine a second area formed by the disparity probability distribution curve and the disparity axis between the maximum and minimum values of different disparities; and determine the confidence of the current disparity based on the first area and the second area.
[0029] According to a third aspect of an embodiment of the present disclosure, there is provided a focusing device, comprising:
[0030] processor;
[0031] a memory for storing processor-executable instructions;
[0032] Wherein, the processor is configured to execute the focusing method described in the first aspect or any one embodiment of the first aspect.
[0033] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the focusing method described in the first aspect or any one of the embodiments of the first aspect.
[0034] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: based on the focus area in the preview screen collected by the terminal, the left view original image and the right view original image corresponding to the focus area are obtained, and the image data source used for focus adjustment is determined. The current disparity between the left view original image and the right view original image is determined, and the disparity probability distribution corresponding to the current disparity is determined. The disparity probability distribution represents the correspondence between multiple different disparities and probability values. The multiple different disparities include the current disparity and multiple disparities with different disparity offsets of the current disparity. The probability value represents the degree of matching between the different disparities and the current disparity, and the current disparity of the focus area is obtained. Based on the disparity probability distribution, the confidence of the current disparity is determined. According to the confidence satisfying the confidence threshold, the current disparity is used for focusing. Through the present disclosure, accurate calculation of the disparity is achieved, a stable and reliable disparity confidence is obtained, the focusing effect in scenes such as dark light or overexposure is improved, the focus loss is reduced, and the user's photo-taking experience is improved.
[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0037] Figure 1 The figure is a flow chart of a focusing method according to an exemplary embodiment.
[0038] Figure 2 The figure is a flow chart showing a method for determining a disparity between a left view and a right view according to an exemplary embodiment.
[0039] Figure 3 The present invention is a schematic diagram showing a process of determining a disparity map of a left view original image and a right view original image according to an exemplary embodiment.
[0040] Figure 4 The figure is a flow chart of a method for calculating a disparity map according to an exemplary embodiment.
[0041] Figure 5 The figure is a flow chart showing a method for calculating the disparity of a target point according to an exemplary embodiment.
[0042] Figure 6 The present invention is a flowchart of a method for determining target point similarity matching according to an exemplary embodiment.
[0043] Figure 7 The present invention is a flowchart of a method for extracting features of a left view original image and a right view original image according to an exemplary embodiment.
[0044] Figure 8 The present invention is a flowchart of a method for predicting disparity by using a depth estimation neural network according to an exemplary embodiment.
[0045] Fig. 9 The figure is a flowchart of a training method for a second network model according to an exemplary embodiment.
[0046] Fig.10 The flowchart of a method for determining current disparity confidence is shown according to an exemplary embodiment.
[0047] Fig.11 is a disparity probability distribution curve shown according to an exemplary embodiment.
[0048] Fig.12 is a disparity probability distribution curve shown according to an exemplary embodiment.
[0049] Fig.13 The figure is a comparison of calculation results of a traditional parallax calculation method and a method of the present disclosure in light and dark environments according to an exemplary embodiment.
[0050] Fig.14 The figure is a flow chart of a focusing method according to an exemplary embodiment.
[0051] Fig.15 The figure is a flow chart of a focusing method according to an exemplary embodiment.
[0052] Fig.16 The figure is a schematic diagram of a multi-window strategy according to an exemplary embodiment.
[0053] Fig.17 is a block diagram of a focusing device 100 according to an exemplary embodiment.
[0054] Fig.18 is a block diagram of a focusing device 200 according to an exemplary embodiment. DETAILED DESCRIPTION
[0055] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0056] The focusing method provided by the embodiments of the present disclosure is applied to scenarios of terminal or camera focusing.
[0057] In recent years, mobile phone cameras have gradually become a must-have for people to take photos during their daily travels, and the performance of mobile phone focus has also become the main direction of competition among major mobile phone manufacturers. At present, taking photos has become the core demand of users to purchase mobile phones, and the new machine release conferences of major manufacturers also use the photo effect as the main selling point. In recent years, PDAF has become the mainstream focus method in the field of mobile phone photography with its advantage of fast focusing speed. PDAF focus mainly obtains left view and right view data by reserving some masked pixels on the photosensitive element as autofocus sensors, and calculates the disparity value by comparing the difference between the left view and the right view pixels. Then, through the distance mapping model, it is converted into the corresponding focus distance, and the lens is moved to the best focus position. As the distance between the lens and the photosensitive element continues to shorten, the phase difference decreases until the phase difference is 0 at the focus point. Subsequently, in the process of continuous shortening, the phase difference changes to the opposite direction, and the absolute value increases. It shows that there is a proportional relationship between the focus distance and the phase difference, and the position of the lens can be reflected according to the current phase difference. At the same time, when this principle is applied based on image data, the parallax between the two views is the phase difference. Therefore, the position of the motor-controlled lens can be reflected by the parallax of the images of the two viewing angles captured by the light-sensing element, so that the movement of the motor-controlled lens can be adjusted according to the parallax, thereby completing the focus of the camera.
[0058] The PDAF focusing method is applicable to a variety of image sensors. On mobile phones, the left view and the right view are mainly captured by Sparse PD sensor (sparse phase difference sensor), Dual PD sensor (dual-core phase difference sensor) or QPD sensor (full pixel phase difference sensor). These two pictures constitute a full-pixel dual-core image pair, which can be regarded as a small baseline stereo image pair with the same exposure time and strict correction. The parallax of the image pair corresponds to the degree of image blur, and there is also parallax in the out-of-focus area. In this technical solution, the calculation of the phase difference between the left view and the right view is particularly critical. Generally, the phase difference PD (Phase Difference) is calculated by using a traditional algorithm. Due to the corresponding relationship between the phase difference and the parallax in the above, and in order to facilitate the visual understanding of the technical solution, PD is used as a representation of the parallax. At present, the calculation of the parallax PD depends on the pixel point being able to obtain sufficiently good luma (brightness) data, and then the sum of the absolute value differences (SAD) of the left view and the right view is calculated to obtain the PD value. The standard Confidence used to evaluate the credibility of the PD value is also calculated by linear correlation with the brightness of the left view and the right view. The higher the Confidence value, the more accurate the PD calculation is, and the lower the Confidence value, the less reliable the PD value is. In a dark or overexposed environment, the confidence of the PD value is greatly reduced, and even the confidence corresponding to the wrong PD may be very high. At the same time, the PD value is also affected by noise, especially in a dark or backlit environment, the PD calculation algorithm is greatly challenged, and inaccurate PD calculation often occurs.
[0059] In order to overcome this problem, the present disclosure proposes a focusing method, based on the left view and the right view, a left view feature map and a right view feature map are obtained through a feature extraction neural network. Based on the left view feature map and the right view feature map, a depth estimation neural network is used to determine the current disparity value and different disparities and their corresponding probability values. The confidence of the current disparity value is obtained through the distribution of different disparities and their corresponding probability values. Based on this disparity value and the corresponding confidence, the camera of the terminal is focused.
[0060] Figure 1 is a flow chart of a focusing method according to an exemplary embodiment. The method is applied to a terminal, such as Figure 1 The following steps are shown:
[0061] In step S11, in response to the preview picture captured by the image capture device in the terminal being triggered to focus, a left view original image and a right view original image corresponding to the focus area are obtained based on the focus area in the preview picture.
[0062] A focus operation is triggered for the preview image captured by the image capture device in the terminal, and the left view original image and the right view original image are obtained from the focus area determined by the focus operation as data sources for adjusting the focus of the focus method.
[0063] In the disclosed embodiment, based on the preview picture captured by the image capture device of the terminal, focus is triggered, and the left view original image and the right view original image are obtained in the focus area of the picture for disparity calculation.
[0064] In step S12, the current disparity between the left view original image and the right view original image is determined, and the disparity probability distribution corresponding to the current disparity is determined, the disparity probability distribution represents the correspondence between multiple different disparities and probability values, the multiple different disparities include the current disparity and multiple disparities with different disparity offsets, and the probability value represents the degree of matching between the different disparities and the current disparity.
[0065] According to the original image of the left view and the original image of the right view, the relationship between the current disparity and the corresponding disparity probability distribution is determined. The disparity probability distribution reflects the corresponding relationship between multiple different disparities and probability values. The multiple different disparities include the current disparity and multiple disparity values with different disparity offsets from the current disparity. The probability value is the degree of matching between different disparity offsets and the current disparity. In this way, the disparity value required for phase focusing can be obtained, and data is provided for the calculation of confidence.
[0066] In the disclosed embodiment, the current disparity and disparity probability distribution are determined through the left view original image and the right view original image, thereby providing disparity data for the focus of the terminal.
[0067] In step S13 , the confidence of the current disparity is determined based on the disparity probability distribution.
[0068] According to the determined current disparity, the disparity probability distribution is obtained by utilizing the probability values determined for the current disparity in different disparity offsets, and the confidence of the current disparity value is determined.
[0069] In the disclosed embodiment, the disparity probability distribution is obtained through the probability values of the current disparity in different disparity offsets, and the confidence of the current disparity value is obtained based on the disparity probability distribution.
[0070] In step S14 , in response to the confidence level satisfying a confidence level threshold, focusing is performed based on the parallax.
[0071] After the confidence of the current disparity value is obtained, the confidence of the current disparity value is compared with a confidence threshold. If the confidence threshold is satisfied, the image acquisition device of the terminal is focused based on the current disparity value.
[0072] In the disclosed embodiment, according to the current parallax and its corresponding confidence level, it is possible to ensure that the camera focuses accurately and reliably.
[0073] In the disclosed embodiment, by extracting the feature maps of the left view and the right view, rich features are provided for the calculation of the disparity. Based on the feature maps of the left view and the right view, different disparities and their corresponding probability values are obtained through the depth estimation neural network, and accurate disparity values of the left view and the right view are obtained. According to the distribution of different disparity probabilities, the confidence of the current disparity is obtained, which provides a reliable confidence for focusing. The accuracy of focusing is ensured, and the user experience in camera focusing is improved.
[0074] In order to achieve an accurate focusing effect, the determination of parallax is particularly important. The embodiment of the present disclosure will hereinafter specifically describe the determination of parallax.
[0075] Figure 2 FIG. 1 is a flow chart showing a method for determining the parallax of a left view and a right view according to an exemplary embodiment. Figure 2 As shown, the determination of the current disparity between the left view and the right view is completed through the following process.
[0076] In step S21a, the image features of the left view original image are extracted to obtain a left view feature image, and the image features of the right view original image are extracted to obtain a right view feature image.
[0077] After obtaining the left view original image and the right view original image, a feature extraction network, such as a convolutional neural network, is used to extract a left view feature image from the left view original image, and a right view feature image from the right view original image to save the extracted image features.
[0078] In step S21b, a first disparity map between the left-view feature image and the right-view feature image is determined, where the first disparity map represents the disparity between feature points in the left-view feature image and the right-view feature image.
[0079] After obtaining the left view feature image and the right view feature image, the disparity of each pixel between the two feature views is obtained according to the depth estimation neural network, and the first disparity map is formed by the disparity of each point corresponding to the two feature images. That is, the first disparity map represents the disparity between the feature points in the left view feature image and the right view feature image.
[0080] In one implementation of this embodiment, Figure 3 is a schematic diagram showing a process of determining a disparity map of a left view original image and a right view original image according to an exemplary embodiment. Figure 3 As shown, the disparity map of the left view original image and the right view original image is determined in the following manner.
[0081] The image data of the left view and the right view are acquired by the phase difference sensor. These two images, namely the full-pixel dual-core image pair, can be regarded as a small baseline stereo image pair with the same exposure time and strict correction. The disparity of the image pair corresponds to the degree of blur of the image, and the disparity exists only in the out-of-focus area. After the camera currently determines the focus area, based on the focus area, the left view original image is obtained from the preview screen, and the right view original image is obtained from the preview screen. The features of the left view original image and the right view original image are extracted through the convolutional neural network, and the features of the two are shared by feature weights. The similarity tensor is further calculated by the convolutional neural network. The similarity between the left view feature map and the right view feature map under different disparity conditions is calculated, and then the disparity is further optimized by the 3D aggregation network. Among them, the 3D aggregation network is a 3D convolutional neural network, which is used to downsample the similarity tensor. Thus, the probability of each feature point on the multi-layer feature map under different disparity conditions is output, and finally a first disparity map with the size of the left view feature map and the right view feature map, namely the depth map, is obtained.
[0082] In step S21c, the first disparity map is upsampled to the original image size to obtain a second disparity map between the left view original image and the right view original image, wherein the second disparity map represents the disparity between pixels in the left view feature image and the right view feature image.
[0083] The first disparity map is upsampled to the size of the left view original image or the right view original image to obtain a second disparity map of the original image size. In this way, the PD value of each pixel point of the original image is obtained, thereby obtaining the second disparity map.
[0084] In step S21d, based on the second disparity map, the current disparity between the left view and the right view is determined.
[0085] According to the second disparity map, the overall disparity value between the left view image and the right view image is obtained by calculating the disparity value corresponding to each pixel point in the second disparity map, for example, finding an average value or a weighted value.
[0086] In the disclosed embodiment, a final disparity value between the left view original image and the right view original image can be obtained through the second disparity map between the two, which is used as the disparity value directly used for focusing.
[0087] In the disclosed embodiment, the second disparity map is obtained by upsampling the first disparity map, and the first disparity map is obtained by feature point by feature point of the left view feature map and the right view feature map.
[0088] Figure 4 FIG. 1 is a flow chart showing a method for calculating a disparity map according to an exemplary embodiment. Figure 4As shown, the disparity map of the left view feature map and the right view feature map is completed through the following process.
[0089] In step S31a, each feature point in the left-view feature image and the right-view feature image is taken as a target feature point, and a first disparity of the target feature point is determined.
[0090] Each feature point of the left-view feature image and the right-view feature image is used as a target feature point, and a first disparity value of the target feature point is determined by epipolar line matching.
[0091] In one implementation of this embodiment, the left view is a reference image and the right view is a to-be-matched image. Under different depth conditions, for example, there are 9 different depths d1, d2, d3, d4, ..., d8, d9, representing the same object point at 9 different depths. The projection point of the same object point in the left and right images is the imaging point. In the reference image, a projected pixel point is referenced. There are 9 corresponding projection pixels in the epipolar line of the to-be-matched image at different depths. According to epipolar line matching, the sequence of projection points at all depths is used as the search space Search Space, and the different projection pixels of the same object point in the reference image and the image to be matched are determined in this Search Space. Thus, the display deviation of the same object in the left and right images is determined, that is, the disparity. For example, there is a reference point (x1, y1) in the reference image. Through epipolar line matching, the corresponding pixel point matched in the to-be-matched image is (x2, y1), then the disparity of the point is deltaX = |x1-x2|, that is, the first disparity of the target feature point of the left view feature map and the right view feature map is deltaX.
[0092] In step S31b, a first disparity map is obtained based on the first disparity of each target feature point.
[0093] According to the obtained first disparity deltaX of each target point, the first disparity value of each target point is determined one by one at the pixel points corresponding to the left view feature map and the right view feature map, so as to obtain a first disparity map.
[0094] In the embodiment of the present disclosure, the method of determining the first disparity deltaX of each target point is applied to the disparity determination of each pixel point in the left view feature map and the right view feature map, so as to obtain the first disparity map.
[0095] The disclosed embodiment determines a first disparity map of a left view feature map and a right view feature map, and the disparity of each pixel in the disparity map is obtained through each matched target feature point.
[0096] Figure 5 FIG. 1 is a flow chart showing a method for calculating the disparity of a target point according to an exemplary embodiment. Figure 5 As shown, the determination of the target feature point disparity is completed through the following process.
[0097] In step S41a, the similarities of the target feature points in the left-view feature image and the right-view feature image under multiple different disparities are determined, and the disparities corresponding to the similarities under multiple different disparities are predicted as probability values of the first disparity.
[0098] Based on the disparity of the target feature point determined in the left view feature image and the right view feature image, a prediction probability value of the point in the feature image pair of the left view and the right view obtained at different lens positions is predicted.
[0099] In one implementation of this embodiment, at different depths, different disparity values are obtained by matching, and the disparity values also correspond to different matching similarities. That is, the horizontal axis is depth, and the vertical axis is Matching cost, which can represent the matching similarity curve of a pixel point in the reference image and 9 pixels in the image to be matched at different depths, which can be represented as a matching cost at the pixel level, and the matching cost at the pixel level constitutes a search space for matching similarities at different depths.
[0100] In one implementation of this embodiment, a feature pixel point in the feature map can constitute a mapping with a dimension of 1 and a total of 9 pixels. For the entire pixel of the feature map, a similarity tensor with a dimension of W*H*9 can be obtained as the search space Search Space, where W is the width of the feature map and H is the height of the feature map.
[0101] In step S41b, the disparity corresponding to the highest probability value is used as the first disparity of the target feature point.
[0102] According to the target feature points of the feature images of the left view and the right view obtained by the predicted different lens positions, based on the probability values of the target feature points at different disparities predicted, the highest probability value is determined as the disparity of the target feature point, that is, the first disparity of the target feature point.
[0103] In the disclosed embodiment, based on the probability values of the target feature point predicted at different disparities, the disparity corresponding to the highest probability value is determined as the disparity value of the target feature point. The probability value is obtained through similarity matching conversion.
[0104] Figure 6 FIG. 1 is a flow chart showing a method for determining target point similarity matching according to an exemplary embodiment. Figure 6 As shown, the similarity of the target feature points under different parallaxes is determined through the following process.
[0105] In step S51a, one of the left-view feature image and the right-view feature image is used as a reference image, and the other is used as an image to be matched.
[0106] One of the left view feature image and the right view feature image is selected as a reference image for target point similarity matching, and the other is selected as a to-be-matched image for target point similarity matching.
[0107] In step S51b, the target feature points in the reference image are mapped to the image to be matched at multiple different parallaxes to obtain multiple target feature points to be matched corresponding to the multiple different parallaxes.
[0108] A point is determined as a target feature point in the reference image, and similarity matching is performed on a pixel sequence in the same row as the target feature point in the image to be matched based on epipolar matching, so that multiple target feature points to be matched with multiple different parallaxes can be obtained.
[0109] In step S51c, the similarity between the target feature point and the multiple target feature points to be matched is used as the similarity of the target feature point under multiple different parallaxes.
[0110] The similarity between the target feature point and multiple target feature points to be matched is evaluated according to the similarity estimation algorithm. For example, SAD can be used to calculate the 3x3 area around the target feature point and the 3x3 area around the target feature point to be matched to obtain the similarity between the two. The similarity is calculated for each target feature point to be matched and used as the similarity of the target feature point under multiple different parallaxes.
[0111] In the disclosed embodiment, by performing similarity matching in the search space to be matched determined by epipolar matching, the similarity of the target feature points under multiple different parallaxes is determined by referring to the target feature points in the image and the target feature points to be matched in the image to be matched.
[0112] The disclosed embodiment determines the similarity of target feature points under different parallaxes, and the determination of the parallax is performed based on the feature image.
[0113] Figure 7 FIG. 1 is a flow chart showing a method for extracting features of a left view original image and a right view original image according to an exemplary embodiment. Figure 7 As shown, the left view feature image and the right view feature image are obtained through the following process.
[0114] In step S61a, the left view original image and the right view original image are respectively input into the first network model, and weight sharing is performed to extract image features of the left view original image and image features of the right view original image.
[0115] The first network model is a feature extraction network, such as a Convolutional Neural Network (CNN), a Visual Geometry Group (VGG) Convolutional Neural Network, a Residual Neural Network (ResNet), etc. When extracting features of the left view original image and the right view original image based on this feature extraction network, weight sharing is performed to extract image features of the left view original image and image features of the right view original image.
[0116] In step S61b, a left-view feature image and a right-view feature image are obtained.
[0117] The image features of the left view original image are used as the left view feature image, and the image features of the right view original image are used as the right view feature image.
[0118] In the disclosed embodiment, features of the left view original image and the right view original image are extracted through a feature extraction neural network to obtain a left view feature image and a right view feature image, which provide rich and reliable feature image data for disparity prediction. Compared with the traditional method of directly relying on the original image for disparity calculation, the disclosed embodiment can provide rich image features, which helps to stabilize the feature extraction in the disparity calculation.
[0119] In the disclosed embodiment, a data source for stable disparity calculation is obtained based on the left view feature image and the right view feature image extracted by the feature extraction neural network, and disparity prediction can be performed based on this data source.
[0120] Figure 8 FIG. 1 is a flow chart showing a method for predicting disparity by using a depth estimation neural network according to an exemplary embodiment. Figure 8 As shown, the prediction of disparity is completed through the following process.
[0121] In step S71a, the target feature points are input into the second network model, the input of the second network model is the feature points, and the output is the disparity.
[0122] After the feature extraction neural network obtains the left view feature image and the right view feature image, the target feature points in the two images are used as the input of the input layer of the second network model. The second network model is a depth estimation neural network, which is used to predict the target feature points corresponding to different disparity categories and what specific disparity they are classified into.
[0123] In step S71b, a first disparity is obtained based on the output of the second network model.
[0124] Through the depth estimation neural network model, the disparity corresponding to the target feature point is predicted as the first disparity.
[0125] In the disclosed embodiment, the disparity values corresponding to the target feature points in the left view feature image and the right view feature image are determined by the second network model. The second network model is trained by the following method.
[0126] Fig. 9 FIG. 1 is a flow chart of a training method for a second network model according to an exemplary embodiment. Fig. 9 As shown, the training of the second network model is completed through the following process.
[0127] In step S81a, an original image sequence is obtained.
[0128] The lens of the terminal's image acquisition device is pushed from the closest point to the farthest point by the motor, or from the farthest point to the closest point, and the image acquisition device continuously acquires images to obtain an original image sequence. The motor's pushing step length is the minimum step length that the motor can push linearly.
[0129] In step S81b, the clearest image in the original image sequence is determined, the position corresponding to the clearest image is obtained, and the disparity map of each image corresponding to the position is determined according to the proportional relationship between the position and the disparity.
[0130] After obtaining the original image sequence, the clearest image is determined from the original image sequence, and the clarity of the image can be evaluated based on a clarity evaluation algorithm, such as the Tenengrad method, the Laplacian gradient method, and the variance method. Thus, the clearest image in the image sequence is determined, and the shooting position corresponding to the clearest image, that is, the depth, is determined. According to the proportional relationship between the position and the disparity, the disparity map of each image corresponding to different shooting positions can be determined.
[0131] In step S81c, a second network model is trained based on the image sequence and the corresponding disparity map.
[0132] According to the disparity map corresponding to the image taken at the determined current depth, the disparity maps of all depths are used as training labels, and the feature images obtained by feature extraction of the images taken at all depths are used as training data. The training sample data set composed of the training data and training labels is used to train the second network model. The loss function can use mean square error (MSE), Smooth L1, etc. to evaluate the distance between the predicted disparity and the real disparity.
[0133] In the disclosed embodiment, a training sample data set of the second network model is generated and trained. When the model is used to predict disparity, the confidence of the current disparity can be determined by the predicted disparity value and the corresponding probability value.
[0134] Fig.10 FIG. 1 is a flow chart of a method for determining a current disparity confidence level according to an exemplary embodiment. Fig.10 As shown, the confidence of the current disparity is determined through the following process.
[0135] In step S91a, based on the disparity probability distribution, a disparity probability distribution curve is determined.
[0136] Through the above second network model, the probability values corresponding to when the current disparity is classified into different disparities are predicted, so as to determine the disparity probability distribution curve.
[0137] In step S91b, the first area is obtained by using the disparity probability distribution curve, with the probability value corresponding to the current disparity value being high and the difference between the maximum value and the minimum value of different disparities being wide.
[0138] Based on the probability values corresponding to different disparities of the disparity probability distribution curve, the probability value corresponding to the current disparity value is high, that is, the highest probability value is determined as high, the difference between the maximum and minimum values of the depths corresponding to all disparities is determined as width, and the area value of the rectangular area formed is calculated as the first area S1.
[0139] In step S91c, a second area formed by the disparity probability distribution curve and the disparity axis between the maximum and minimum values of different disparities is determined.
[0140] The area formed by the disparity probability distribution curve and the disparity axis within all disparities is the second area S2.
[0141] In step S91d, the confidence of the current disparity is determined according to the first area and the second area.
[0142] According to the first area S1 and the second area S2, the confidence of the current disparity is Confidence=S1 / S2.
[0143] In one implementation of the present disclosure, Fig.11 The disparity probability distribution curve shown in FIG. 1 takes the probability value corresponding to the current disparity value as the height h=0.47, and the difference between the maximum and minimum values of different disparities as the width w=580-20=560, thereby obtaining an area S1=h*w=0.47*560=263.2 of a rectangular area as the first area S1=263.2, and according to the weighted sum of each disparity and the corresponding depth, that is, by S2 can be obtained, where Dmax represents the maximum disparity, k represents different disparities, and pk represents the probability under the disparity. In this example, S2 = 150, so the confidence of the disparity is Confidence1 = S1 / S2 = 263.2 / 150 = 1.75.
[0144] In one implementation of the present disclosure, Fig.12 A disparity probability distribution curve is shown, in this example, the first area S1 = 0.83*(595-5) = 489.7, the second area S2 = 93. The confidence of the disparity is Confidence2 = S1 / S = 489.7 / 93 = 5.27.
[0145] According to the comparison of the above two implementations, Confidence1 is smaller than Confidence2. It is obvious that the probability value of the current disparity in Confidence2 is higher, and the probability of the disparity adjacent to the current disparity also presents a higher probability value. Therefore, the confidence Confidence2 is more reliable than Confidence1.
[0146] In one implementation of the present disclosure, Fig.13 The comparison of the calculation results of the traditional parallax calculation method and the method disclosed in the present invention in bright and dark environments is shown in FIG. Fig.13 As shown, the horizontal axis is the position of the lens, and the vertical axis is the calculated value of PD. 1000lux_TE42__pd represents the curve of PD value and lens position calculated by the traditional algorithm in an environment with a brightness of 1000lux; 1000lux_TE42_mlpd represents the curve of PD value and lens position calculated by the technical solution of the present disclosure in an environment with a brightness of 1000lux; 1lux_TE42_pd represents the curve of PD value and lens position calculated by the traditional algorithm in an environment with a brightness of 1lux; 1lux_TE42_mlpd represents the curve of PD value and lens position calculated by the technical solution of the present disclosure in an environment with a brightness of 1lux. That is, in a dark light environment with an illumination of 1lux, the relationship curve between the lens position cur_lens_pos and the parallax pd obtained by the traditional PD algorithm fluctuates greatly, and does not reflect the relationship between the two. However, by adopting the technical solution proposed in the present disclosure, a nearly linear relationship between cur_lens_pos and pd is obtained, which shows the excellent focusing performance of the technical solution of the present disclosure in a dark light environment. In a backlit environment with an illumination of 1000 lux, the traditional PD algorithm cannot reflect the linear relationship between cur_lens_pos and pd at many lens positions. However, the technical solution disclosed in the present invention demonstrates a very accurate linear relationship between cur_lens_pos and pd in a backlit environment, achieving very excellent performance.
[0147] In one implementation of this embodiment, for example, the confidence threshold is set to 4. When the confidence Confidence is greater than 4, it is determined that the accuracy of the parallax predicted at this time is high, and the motor can be moved by a specified number of step units according to the current predicted parallax value to adjust the lens to the focus position. If the confidence Confidence is less than or equal to 4, it is determined that the accuracy of the parallax predicted at this time is low, which is not enough to ensure that the current predicted parallax value is an accurate result, and the motor is abandoned to adjust the lens position.
[0148] Fig.14 FIG. 1 is a flow chart of a focusing method according to an exemplary embodiment. Fig.14 As shown, the camera starts taking pictures, the exposure is completed, and the left view preview image and the right view preview image are obtained. According to the focus area, the left view original image and the right view original image are obtained. Based on the left view original image and the right view original image, the image features are extracted through the convolutional neural network, and the feature image is input into the decision network to predict the type of pixel offset, that is, the disparity category. The pixel offset corresponding to the largest predicted value in the pixel offset class is taken as the disparity PD value. According to the curve of the multi-category value of the pixel offset and the corresponding probability value, the confidence Confidence is calculated. According to the Confidence threshold, it is judged whether the disparity PD is credible. If it is credible, the motor is moved according to the PD value to focus.
[0149] Fig.15 FIG. 1 is a flow chart of a focusing method according to an exemplary embodiment. Fig.15As shown, the camera is turned on, exposure is completed, and a preview image of the left view and a preview image of the right view are obtained. According to the focus area, an effective original image of the left view and an original image of the right view are obtained, wherein the focus area can be determined by center focus, touch focus, face focus, eye focus, etc. After obtaining the original image of the left view and the original image of the right view, disparity estimation can be performed. Among them, disparity estimation includes extracting image features of the original image of the left view and the original image of the right view through a convolution neural network, and performing weight sharing in the process of extracting features. After feature extraction, a feature image of the left view and a feature image of the right view are obtained. According to the feature image of the left view and the feature image of the right view, a similarity tensor composed of the similarity of each feature point of the two images is calculated based on epipolar matching, and then the three-dimensional tensor of the similarity tensor is downsampled by a three-dimensional convolution calculation through a three-dimensional aggregation network. The similarity is converted into a probability value for different disparity classifications through a multi-classification network, and the disparity corresponding to the highest probability value is determined as the accurate disparity value at the point, thereby obtaining a disparity map between the feature image of the left view and the feature image of the right view. Each pixel in the disparity map contains the disparity value at that point, and the size of the disparity map is the same as the left view feature map and the right view feature map. Then the disparity map is upsampled to the same size as the left view original map and the right view original map. The upsampling can be completed by deconvolution operation. According to the above processing, the disparity map corresponding to the left view original map and the right view original map can be determined.
[0150] According to the focus area, the region of interest (ROI) in the disparity map is determined, and the disparity value and the corresponding confidence value in this region are calculated. The ROI here can be determined by a multi-window strategy, such as Fig.16 is a multi-window strategy according to an exemplary embodiment. Fig.16 FIG. 1 is a schematic diagram of a multi-window strategy according to an exemplary embodiment. Fig.16 As shown, the central area of the preview screen can be determined as the ROI, or the area determined by detecting the face area, the eye area, etc. can be used as the ROI.
[0151] The parallax value with the highest probability is selected from different parallax values as the parallax value of the adjustable motor, and the confidence of the parallax value is determined. If the confidence meets the confidence threshold, the motor can be translated by using the current credible parallax PD value and converted into the driving distance of the motor.
[0152] Thus completing the focusing.
[0153] Based on the same concept, the embodiment of the present disclosure further provides a focusing device 100 .
[0154] It is understandable that the focusing device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.
[0155] Fig.17 is a block diagram of a focusing device 100 according to an exemplary embodiment. Fig.17 The device includes a collection unit 101, a disparity determination unit 102, a confidence determination unit 103 and a focusing unit 104.
[0156] The acquisition unit 101 is configured to obtain a left view original image and a right view original image corresponding to the focus area based on the focus area in the preview image in response to the preview image acquired by the image acquisition device being triggered to perform focusing.
[0157] The disparity determination unit 102 is configured to determine the current disparity between the left view original image and the right view original image, and determine the disparity probability distribution corresponding to the current disparity, wherein the disparity probability distribution represents the correspondence between multiple different disparities and probability values, wherein the multiple different disparities include the current disparity and multiple disparities in which the current disparity has different disparity offsets, and the probability value represents the degree of matching between the different disparities and the current disparity.
[0158] The confidence determination unit 103 is configured to determine the confidence of the current disparity based on the disparity probability distribution.
[0159] The focusing unit 104 is configured to perform focusing based on the parallax in response to the confidence level satisfying a confidence level threshold.
[0160] In one implementation, the disparity determining unit 102 determines the current disparity between the left view original image and the right view original image in the following manner:
[0161] Extracting image features of the left view original image to obtain a left view feature image, and extracting image features of the right view original image to obtain a right view feature image;
[0162] Determine a first disparity map between the left-view feature image and the right-view feature image, wherein the first disparity map represents a disparity between feature points in the left-view feature image and the right-view feature image;
[0163] Upsampling the first disparity map to an original image size to obtain a second disparity map between the left-view original image and the right-view original image, wherein the second disparity map represents a disparity between pixels in the left-view feature image and the right-view feature image;
[0164] Based on the second disparity map, a current disparity between the left view and the right view is determined.
[0165] In one implementation, the disparity determining unit 102 determines the first disparity map between the left view feature image and the right view feature image in the following manner:
[0166] For each feature point in the left view feature image and the right view feature image, respectively, take it as a target feature point and determine a first disparity of the target feature point;
[0167] Based on the first disparity of each of the target feature points, a first disparity map is obtained;
[0168] Wherein, determining the first disparity of the target feature point includes:
[0169] Determine similarities of target feature points in the left view feature image and the right view feature image under multiple different disparities, and predict the probability values of the disparities corresponding to the similarities under the multiple different disparities as the first disparity;
[0170] The disparity corresponding to the highest probability value is used as the first disparity of the target feature point.
[0171] In one implementation, the disparity determination unit 102 determines the similarity of the target feature points in the left view feature image and the right view feature image under multiple different disparities in the following manner:
[0172] Using one of the left view feature image and the right view feature image as a reference image and the other as an image to be matched;
[0173] Mapping the target feature points in the reference image to the image to be matched at multiple different parallaxes to obtain multiple target feature points to be matched corresponding to the multiple different parallaxes;
[0174] The similarity between the target feature point and the multiple to-be-matched target feature points is used as the similarity of the target feature point under multiple different parallaxes.
[0175] In one implementation manner, the disparity determining unit 102 is further configured to determine the disparity probability distribution based on the first disparity and the predicted similarity corresponding disparities under the plurality of different disparities as a probability value of the first disparity.
[0176] In one implementation, the disparity determination unit 102 extracts image features of the left view original image to obtain a left view feature image, and extracts image features of the right view original image to obtain a right view feature image in the following manner:
[0177] Inputting the left view original image and the right view original image into a first network model respectively, and performing weight sharing to extract image features of the left view original image and image features of the right view original image, so as to obtain a left view feature image and a right view feature image;
[0178] The input of the first network model is an image, and the output is image features.
[0179] In one implementation, the disparity determination unit 102 determines the first disparity of the target feature point in the following manner:
[0180] Inputting the target feature point into a second network model, wherein the input of the second network model is the feature point and the output is the disparity;
[0181] Based on the output of the second network model, a first disparity is obtained.
[0182] In one implementation, the disparity determination unit 102 trains the second network model in the following manner:
[0183] Obtaining the original image sequence;
[0184] Determine the clearest image from the original image sequence, obtain a position corresponding to the clearest image, and determine a disparity map of each image corresponding to the position according to a proportional relationship between the position and the disparity;
[0185] The second network model is trained based on the image sequence and the corresponding disparity map.
[0186] In one implementation, the confidence determination unit 103 determines the confidence of the current disparity based on the disparity probability distribution in the following manner:
[0187] Based on the disparity probability distribution, determining the disparity probability distribution curve;
[0188] According to the disparity probability distribution curve, a first area is obtained by taking the probability value corresponding to the current disparity value as high and the difference between the maximum value and the minimum value of the different disparities as wide;
[0189] A second area formed by the disparity probability distribution curve and the disparity axis between the maximum and minimum values of different disparities is determined.
[0190] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0191] Fig.18 2 is a block diagram of a focusing apparatus 200 according to an exemplary embodiment. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0192] Reference Fig.18 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .
[0193] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.
[0194] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0195] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.
[0196] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0197] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.
[0198] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0199] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.
[0200] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0201] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0202] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0203] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.
[0204] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.
[0205] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.
[0206] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.
[0207] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.
[0208] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.
[0209] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.
Claims
1. A focusing method, characterized in that: Applied to terminals, including: In response to the preview picture captured by the image capture device in the terminal being triggered to focus, obtaining a left view original image and a right view original image corresponding to the focus area based on the focus area in the preview picture; Determine a current disparity between the left view original image and the right view original image, and determine a disparity probability distribution corresponding to the current disparity, wherein the disparity probability distribution represents a correspondence between a plurality of different disparities and probability values, wherein the plurality of different disparities include the current disparity and a plurality of disparities having different disparity offsets of the current disparity, and the probability value represents a matching degree between the different disparities and the current disparity; Determining a confidence level of the current disparity based on the disparity probability distribution; In response to the confidence satisfying a confidence threshold, focusing is performed based on the disparity.
2. The method according to claim 1, characterized in that The determining a current disparity between the left-view original image and the right-view original image comprises: Extracting image features of the left view original image to obtain a left view feature image, and extracting image features of the right view original image to obtain a right view feature image; Determine a first disparity map between the left-view feature image and the right-view feature image, wherein the first disparity map represents a disparity between feature points in the left-view feature image and the right-view feature image; Upsampling the first disparity map to an original image size to obtain a second disparity map between the left-view original image and the right-view original image, wherein the second disparity map represents the disparity between pixels in the left-view feature image and the right-view feature image; Based on the second disparity map, a current disparity between the left view and the right view is determined.
3. The method according to claim 2, characterized in that The determining a first disparity map between the left view feature image and the right view feature image comprises: For each feature point in the left view feature image and the right view feature image, respectively, take it as a target feature point and determine a first disparity of the target feature point; Based on the first disparity of each of the target feature points, a first disparity map is obtained; Wherein, determining the first disparity of the target feature point includes: Determine similarities of target feature points in the left view feature image and the right view feature image under multiple different disparities, and predict the probability values of the disparities corresponding to the similarities under the multiple different disparities as the first disparity; The disparity corresponding to the highest probability value is used as the first disparity of the target feature point.
4. The method according to claim 3, characterized in that: The determining of the similarity of the target feature points in the left view feature image and the right view feature image under a plurality of different parallaxes comprises: Using one of the left view feature image and the right view feature image as a reference image and the other as an image to be matched; Mapping the target feature points in the reference image to the image to be matched at multiple different parallaxes to obtain multiple target feature points to be matched corresponding to the multiple different parallaxes; The similarity between the target feature point and the multiple to-be-matched target feature points is used as the similarity of the target feature point under multiple different parallaxes.
5. The method according to claim 3 or 4, characterized in that: The disparity probability distribution is determined based on the first disparity and the predicted probability value of the disparity corresponding to the similarity under the plurality of different disparities as the first disparity.
6. The method according to claim 2, characterized in that The extracting the image features of the left view original image to obtain the left view feature image, and extracting the image features of the right view original image to obtain the right view feature image, comprises: Inputting the left view original image and the right view original image into a first network model respectively, and performing weight sharing to extract image features of the left view original image and image features of the right view original image, so as to obtain a left view feature image and a right view feature image; The input of the first network model is an image, and the output is image features.
7. The method according to claim 2 or 3, characterized in that: The determining the first disparity of the target feature point includes: Inputting the target feature point into a second network model, wherein the input of the second network model is the feature point and the output is the disparity; Based on the output of the second network model, a first disparity is obtained.
8. The method according to claim 7, characterized in that The second network model is trained in the following way: Obtaining the original image sequence; Determine the clearest image from the original image sequence, obtain a position corresponding to the clearest image, and determine a disparity map of each image corresponding to the position according to a proportional relationship between the position and the disparity; The second network model is trained based on the image sequence and the corresponding disparity map.
9. The method according to claim 1, characterized in that: The determining the confidence of the current disparity based on the disparity probability distribution includes: Based on the disparity probability distribution, determining the disparity probability distribution curve; According to the disparity probability distribution curve, a first area is obtained by taking the probability value corresponding to the current disparity value as high and the difference between the maximum value and the minimum value of the different disparities as wide; Determine a second area formed by the disparity probability distribution curve and the disparity axis between the maximum and minimum values of different disparities; The confidence level of the current disparity is determined according to the first area and the second area.
10. A focusing device, characterized in that: The focusing device comprises: A capture unit, configured to, in response to a preview image captured by an image capture device being triggered to focus, obtain a left view original image and a right view original image corresponding to the focus area based on the focus area in the preview image; a disparity determination unit, configured to determine a current disparity between the left view original image and the right view original image, and determine a disparity probability distribution corresponding to the current disparity, wherein the disparity probability distribution represents a correspondence between a plurality of different disparities and probability values, wherein the plurality of different disparities include the current disparity and a plurality of disparities having different disparity offsets from the current disparity, and the probability value represents a matching degree between the different disparities and the current disparity; a confidence determination unit, configured to determine the confidence of the current disparity based on the disparity probability distribution; A focusing unit is configured to focus based on the parallax in response to the confidence level satisfying a confidence level threshold.
11. A focusing device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the focusing method described in any one of claims 1 to 9.
12. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the focusing method described in any one of claims 1 to 9.
Citation Information
Patent Citations
Quick focusing method and electronic equipment
CN105744138A
Camera device and focusing method
CN111345025A
Image processing method, image processing apparatus, image capturing apparatus and non-transitory computer-readable storage medium
US20150163479A1
Depth estimation method and apparatus for binocular image, and device, program and medium
WO2020019765A1
