Focusing method, focusing device and storage medium

Abstract

This disclosure relates to a focusing method, apparatus, and storage medium. The focusing method includes: triggering focusing in response to a preview image captured by an image acquisition device in a terminal; obtaining a left-view original image and a right-view original image corresponding to the focusing area based on the focusing area in the preview image; determining the current parallax between the left-view original image and the right-view original image, and determining the parallax probability distribution corresponding to the current parallax; determining the confidence level of the current parallax based on the parallax probability distribution; and focusing based on the parallax in response to the confidence level meeting a confidence level threshold. This disclosure solves the problem of inaccurate parallax and confidence level calculations, improves focusing performance in low-light or overexposed scenarios, reduces focus loss, and enhances the user's photography experience.

Description

Technical Field

[0001] This disclosure relates to the field of cameras, and more particularly to focusing methods, focusing devices, and storage media. Background Technology

[0002] In related technologies, the calculation of parallax is crucial in the process of phase difference autofocus (PDAF), which directly affects whether the focus is clear. The parallax calculated by the relevant algorithms in low light or overexposed environments, as well as the confidence level of the parallax, cannot accurately reflect the real and stable parallax, resulting in the camera being unable to focus accurately. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, this disclosure provides a focusing method, a focusing device, and a storage medium.

[0004] According to a first aspect of the present disclosure, a focusing method is provided, applied to a terminal, the method comprising:

[0005] In response to the preview screen captured by the image acquisition device in the terminal being triggered to focus, the original left view image and the original right view image corresponding to the focus area are obtained based on the focus area in the preview screen.

[0006] The current disparity between the original left view image and the original right view image is determined, and the disparity probability distribution corresponding to the current disparity is determined. The disparity probability distribution represents the correspondence between multiple different disparities and probability values. The multiple different disparities include the current disparity and multiple disparities with different disparity offsets. The probability values ​​represent the degree of matching between the different disparities and the current disparity. Based on the disparity probability distribution, the confidence level of the current disparity is determined. In response to the confidence level satisfying a confidence threshold, focusing is performed based on the disparity.

[0007] In one embodiment, determining the current disparity between the original left-view image and the original right-view image includes: extracting image features from the original left-view image to obtain a left-view feature image, and extracting image features from the original right-view image to obtain a right-view feature image; determining a first disparity map between the left-view feature image and the right-view feature image, wherein the first disparity map represents the disparity between feature points in the left-view feature image and the right-view feature image; upsampling the first disparity map to the original image size to obtain a second disparity map between the original left-view image and the original right-view image, wherein the second disparity map represents the disparity between pixels in the left-view feature image and the right-view feature image; and determining the current disparity between the left-view and right-view images based on the second disparity map.

[0008] In one embodiment, determining the first disparity map between the left-view feature image and the right-view feature image includes: for each feature point in the left-view feature image and the right-view feature image, respectively, taking it as a target feature point, determining the first disparity of the target feature point; and obtaining a first disparity map based on the first disparity of each target feature point; wherein, determining the first disparity of the target feature point includes: determining the similarity of the target feature points in the left-view feature image and the right-view feature image under multiple different disparities, and predicting the probability value of the disparity corresponding to the similarity under the multiple different disparities as the first disparity; and taking the disparity corresponding to the highest probability value as the first disparity of the target feature point.

[0009] In one embodiment, determining the similarity of target feature points in the left view feature image and the right view feature image under multiple different disparities includes: using one of the left view feature image and the right view feature image as a reference image and the other as a matching image; mapping the target feature points in the reference image to the matching image under multiple different disparities to obtain multiple matching target feature points corresponding to the multiple different disparities; and using the similarity between the target feature points and the multiple matching target feature points as the similarity of the target feature points under multiple different disparities.

[0010] In one embodiment, the disparity probability distribution is determined based on the first disparity and the probability value of predicting the disparity corresponding to the similarity under the plurality of different disparities as the first disparity.

[0011] In one embodiment, the step of extracting image features from the original left-view image to obtain a left-view feature image, and extracting image features from the original right-view image to obtain a right-view feature image, includes: inputting the original left-view image and the original right-view image into a first network model respectively, and performing weight sharing to extract image features from the original left-view image and the original right-view image to obtain a left-view feature image and a right-view feature image; the input of the first network model is an image, and the output is image features.

[0012] In one embodiment, determining the first disparity of the target feature point includes: inputting the target feature point into a second network model, wherein the input of the second network model is the feature point and the output is the disparity; and obtaining the first disparity based on the output of the second network model.

[0013] In one embodiment, the second network model is trained as follows: obtaining an original image sequence; determining the clearest image from the original image sequence, obtaining the position corresponding to the clearest image, and determining the disparity map of each image corresponding to the position based on the ratio between the position and the disparity; and training the second network model based on the image sequence and the corresponding disparity map.

[0014] In one embodiment, determining the confidence level of the current disparity based on the disparity probability distribution includes: determining the disparity probability distribution curve based on the disparity probability distribution; obtaining a first area by using the disparity probability distribution curve with the probability value corresponding to the current disparity value as high and the difference between the maximum and minimum values ​​of different disparities as wide; determining a second area formed by the disparity probability distribution curve and the disparity axis between the maximum and minimum values ​​of different disparities; and determining the confidence level of the current disparity based on the first area and the second area.

[0015] According to a second aspect of the present disclosure, a focusing device is provided, applied to a terminal, the focusing device comprising:

[0016] The acquisition unit is used to focus in response to the preview screen acquired by the image acquisition device being triggered, and to obtain the original left view image and the original right view image corresponding to the focus area based on the focus area in the preview screen.

[0017] The disparity determination unit is used to determine the current disparity between the original left view image and the original right view image, and to determine the disparity probability distribution corresponding to the current disparity. The disparity probability distribution characterizes the correspondence between multiple different disparities and probability values. The multiple different disparities include the current disparity and multiple disparities with different disparity offsets. The probability values ​​characterize the degree of matching between the different disparities and the current disparity.

[0018] A confidence determination unit is used to determine the confidence level of the current disparity based on the disparity probability distribution.

[0019] A focusing unit is configured to focus based on the parallax in response to the confidence level meeting a confidence threshold.

[0020] In one embodiment, the disparity determination unit determines the current disparity between the original left-view image and the original right-view image in the following manner: extracting image features from the original left-view image to obtain a left-view feature image, and extracting image features from the original right-view image to obtain a right-view feature image; determining a first disparity map between the left-view feature image and the right-view feature image, wherein the first disparity map represents the disparity between feature points in the left-view feature image and the right-view feature image; upsampling the first disparity map to the original image size to obtain a second disparity map between the original left-view image and the original right-view image, wherein the second disparity map represents the disparity between pixels in the left-view feature image and the right-view feature image; and determining the current disparity between the left-view and right-view images based on the second disparity map.

[0021] In one embodiment, the disparity determination unit determines a first disparity map between the left-view feature image and the right-view feature image in the following manner: for each feature point in the left-view feature image and the right-view feature image, respectively, as a target feature point, a first disparity of the target feature point is determined; based on the first disparity of each target feature point, a first disparity map is obtained; wherein, determining the first disparity of the target feature point includes: determining the similarity of the target feature points in the left-view feature image and the right-view feature image under multiple different disparities, and predicting the probability value of the disparity corresponding to the similarity under the multiple different disparities as the first disparity; taking the disparity corresponding to the highest probability value as the first disparity of the target feature point.

[0022] In one embodiment, the disparity determination unit determines the similarity of target feature points in the left view feature image and the right view feature image under multiple different disparities in the following manner: one of the left view feature image and the right view feature image is used as a reference image, and the other is used as a matching image; the target feature points in the reference image are mapped to the matching image under multiple different disparities to obtain multiple matching target feature points corresponding to the multiple different disparities; the similarity between the target feature points and the multiple matching target feature points is used as the similarity of the target feature points under multiple different disparities.

[0023] In one embodiment, the disparity determination unit is further configured to determine the disparity probability distribution based on the first disparity and the probability value of the predicted disparity corresponding to the similarity under the plurality of different disparities as the first disparity.

[0024] In one embodiment, the disparity determination unit extracts image features from the original left view image to obtain a left view feature image, and extracts image features from the original right view image to obtain a right view feature image:

[0025] The original left-view image and the original right-view image are respectively input into the first network model and weights are shared to extract the image features of the original left-view image and the original right-view image, resulting in a left-view feature image and a right-view feature image; the input of the first network model is an image, and the output is image features.

[0026] In one embodiment, the disparity determination unit determines the first disparity of the target feature point in the following manner: the target feature point is input into a second network model, the input of the second network model is the feature point, and the output is the disparity; the first disparity is obtained based on the output of the second network model.

[0027] In one embodiment, the disparity determination unit trains the second network model in the following manner: obtaining an original image sequence; determining the clearest image from the original image sequence, obtaining the position corresponding to the clearest image, and determining the disparity map of each image corresponding to the position based on the ratio between the position and the disparity; and training the second network model based on the image sequence and the corresponding disparity map.

[0028] In one embodiment, the confidence determination unit determines the confidence of the current disparity based on the disparity probability distribution in the following manner: Based on the disparity probability distribution, a disparity probability distribution curve is determined; using the disparity probability distribution curve, a first area is obtained with the probability value corresponding to the current disparity value as high and the difference between the maximum and minimum values ​​of different disparities as wide; a second area is determined between the maximum and minimum values ​​of different disparities formed by the disparity probability distribution curve and the disparity axis; and the confidence of the current disparity is determined based on the first area and the second area.

[0029] According to a third aspect of the present disclosure, a focusing device is provided, comprising:

[0030] processor;

[0031] Memory used to store processor-executable instructions;

[0032] The processor is configured to execute the focusing method described in the first aspect or any embodiment of the first aspect.

[0033] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor of a terminal, enable the terminal to perform the focusing method described in the first aspect or any embodiment of the first aspect.

[0034] The technical solution provided in this disclosure includes the following beneficial effects: Based on the focus area in the preview image captured by the terminal, the original left-view image and the original right-view image corresponding to the focus area are obtained, and the image data source used for focus adjustment is determined. The current disparity between the original left-view image and the original right-view image is determined, and the disparity probability distribution corresponding to the current disparity is determined. The disparity probability distribution represents the correspondence between multiple different disparities and probability values. Among the multiple different disparities, there are the current disparity and multiple disparities with different disparity offsets. The probability value represents the degree of matching between different disparities and the current disparity, and the current disparity of the focus area is obtained. Based on the disparity probability distribution, the confidence level of the current disparity is determined. According to the confidence level satisfying the confidence level threshold, focus is performed using the current disparity. Through this disclosure, accurate calculation of disparity is achieved, a stable and reliable disparity confidence level is obtained, the focusing effect in low light or overexposure scenarios is improved, focus loss is reduced, and the user's photography experience is improved.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0037] Figure 1 This is a flowchart illustrating a focusing method according to an exemplary embodiment.

[0038] Figure 2 This is a flowchart illustrating a method for determining the parallax of a left view and a right view according to an exemplary embodiment.

[0039] Figure 3 This is a schematic diagram illustrating a process for determining a parallax map of an original left-view image and an original right-view image, according to an exemplary embodiment.

[0040] Figure 4 This is a flowchart illustrating a disparity map calculation method according to an exemplary embodiment.

[0041] Figure 5 This is a flowchart illustrating a method for calculating the disparity of a target point according to an exemplary embodiment.

[0042] Figure 6 This is a flowchart illustrating a method for determining target point similarity matching according to an exemplary embodiment.

[0043] Figure 7 This is a flowchart illustrating a feature extraction method for a left-view original image and a right-view original image according to an exemplary embodiment.

[0044] Figure 8 This is a flowchart illustrating a method for predicting disparity using a depth estimation neural network according to an exemplary embodiment.

[0045] Figure 9 This is a flowchart illustrating a training method for a second network model according to an exemplary embodiment.

[0046] Figure 10 This is a flowchart illustrating a method for determining the current disparity confidence level according to an exemplary embodiment.

[0047] Figure 11 This is a disparity probability distribution curve illustrated according to an exemplary embodiment.

[0048] Figure 12 This is a disparity probability distribution curve illustrated according to an exemplary embodiment.

[0049] Figure 13 This is a comparison of the calculation results of a conventional parallax calculation method and the method of this disclosure under bright and dark environments, according to an exemplary embodiment.

[0050] Figure 14 This is a flowchart illustrating a focusing method according to an exemplary embodiment.

[0051] Figure 15 This is a flowchart illustrating a focusing method according to an exemplary embodiment.

[0052] Figure 16 This is a schematic diagram illustrating a multi-window strategy according to an exemplary embodiment.

[0053] Figure 17 This is a block diagram illustrating a focusing device 100 according to an exemplary embodiment.

[0054] Figure 18 This is a block diagram illustrating a focusing device 200 according to an exemplary embodiment. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0056] The focusing method provided in this disclosure is applied to scenarios where a terminal or camera is focusing.

[0057] In recent years, mobile phone cameras have gradually become essential for people's daily travel and photography, and the focusing performance of mobile phones has become a major area of ​​competition among mobile phone manufacturers. Currently, photography has become a core requirement for users when purchasing mobile phones, and major manufacturers use photo quality as a primary selling point at new phone launches. In recent years, PDAF (PDental autofocus) has become the mainstream focusing method in the field of mobile phone photography due to its fast focusing speed. PDAF focusing mainly involves reserving some masked pixels on the image sensor to act as an autofocus sensor, obtaining left and right view data. By comparing the differences between the pixels in the left and right views, the parallax value is calculated. Then, through a distance mapping model, it is converted into the corresponding focusing distance, moving the lens to the optimal focusing position. As the distance between the lens and the image sensor continuously shortens, the phase difference decreases until it reaches zero at the point of focus. Subsequently, as the distance continues to shorten, the phase difference reverses direction, and its absolute value increases. This indicates that there is a proportional relationship between the focusing distance and the phase difference, allowing the lens position to be reflected based on the current phase difference. Furthermore, when this principle is applied based on image data, the parallax between the two views is the phase difference. Therefore, the parallax of the two images captured by the light sensor can reflect the position of the motor-controlled lens, and the movement of the lens controlled by the motor can be adjusted according to the parallax, thereby completing the camera's focusing.

[0058] PDAF focusing methods are applicable to various image sensors. In mobile phones, they are mainly captured using sparse PD sensors, dual PD sensors, or QPD sensors to capture left and right views. These two images constitute a full-pixel dual-core image pair, which can be viewed as a small-baseline stereo image pair with the same exposure time and strict correction. The parallax of this image pair corresponds to the degree of image blur, and parallax also exists in the out-of-focus area. In this technical solution, the calculation of the phase difference between the left and right views is particularly critical. Generally, the phase difference (PD) is calculated using traditional algorithms. Due to the correspondence between phase difference and parallax mentioned above, and for the sake of easier understanding of the technical solution, PD is used as a representation of parallax. Currently, parallax PD calculation relies on obtaining sufficiently good luma (brightness) data for each pixel, and then calculating the sum of absolute differences (SAD) between the left and right views to obtain the PD value. The confidence level, used to evaluate the reliability of PD values, is calculated by linearly correlating it with the brightness of the left and right views. A higher confidence value indicates a more accurate PD calculation, while a lower confidence value indicates a less reliable PD value. In low-light or overexposed environments, the confidence level of PD values ​​decreases significantly, and in some cases, a high confidence level may correspond to an incorrect PD value. Furthermore, PD values ​​are also affected by noise, especially in low-light or backlit environments, where the PD calculation algorithm faces significant challenges and frequently results in inaccurate PD calculations.

[0059] To overcome this problem, this disclosure proposes a focusing method that, based on left and right views, obtains feature maps for the left and right views using a feature extraction neural network. Based on these feature maps, a depth estimation neural network is used to determine the current disparity value and different disparities and their corresponding probability values. The confidence level of the current disparity value is obtained through the distribution of different disparities and their corresponding probability values. Based on this disparity value and the corresponding confidence level, the terminal's camera is focused.

[0060] Figure 1 This is a flowchart illustrating a focusing method according to an exemplary embodiment. The method is applied to a terminal, such as... Figure 1 The steps shown are as follows:

[0061] In step S11, focusing is triggered in response to the preview screen captured by the image acquisition device in the terminal, and the original left view image and the original right view image corresponding to the focusing area are obtained based on the focusing area in the preview screen.

[0062] The preview image captured by the image acquisition device in the terminal is triggered to perform a focusing operation. The original images of the left and right views are obtained from the focusing area determined by the focusing operation, and used as the data source for adjusting the focus in this focusing method.

[0063] In this embodiment of the disclosure, the preview image captured by the terminal's image acquisition device is used to trigger focusing, and the original left view image and the original right view image are obtained in the focused area of ​​the image for parallax calculation.

[0064] In step S12, the current disparity between the original left view image and the original right view image is determined, and the disparity probability distribution corresponding to the current disparity is determined. The disparity probability distribution represents the correspondence between multiple different disparities and probability values. Among the multiple different disparities, there are the current disparity and multiple disparities with different disparity offsets. The probability value represents the degree of matching between different disparities and the current disparity.

[0065] Based on the original images from the left and right views, the relationship between the current disparity and its corresponding disparity probability distribution is determined. The disparity probability distribution reflects the correspondence between multiple different disparities and their probability values. These multiple different disparities include the current disparity and multiple disparity values ​​with different disparity offsets. The probability value represents the degree of matching between different disparity offsets and the current disparity. This allows us to obtain the disparity values ​​required for phase-detection autofocus and provides data for confidence level calculation.

[0066] In this embodiment of the disclosure, the current disparity and disparity probability distribution are determined by the original left view image and the original right view image, providing disparity data for the terminal's focusing.

[0067] In step S13, the confidence level of the current disparity is determined based on the disparity probability distribution.

[0068] Based on the determined current disparity, the probability distribution of the disparity is obtained by using the probability values ​​of the current disparity in different disparity offsets, and the confidence level of the current disparity value is determined.

[0069] In this embodiment of the disclosure, a disparity probability distribution is obtained by using the probability values ​​of the current disparity in different disparity offsets. Based on this disparity probability distribution, the confidence level of the current disparity value is obtained.

[0070] In step S14, focusing is performed based on parallax in response to the confidence level meeting the confidence threshold.

[0071] After obtaining the confidence level of the current disparity value, the confidence level of the current disparity value is compared with the confidence threshold. If the confidence threshold is met, the image acquisition device of the terminal is focused based on the current disparity value.

[0072] In this embodiment of the disclosure, based on the current parallax and its corresponding confidence level, it is possible to ensure that the camera focuses accurately and reliably.

[0073] In this embodiment, feature maps of the left and right views are extracted, providing rich features for disparity calculation. Based on the feature maps of the left and right views, a deep estimation neural network is used to obtain different disparities and their corresponding probability values, resulting in accurate disparity values ​​for the left and right views. According to the distribution of different disparity probabilities, the confidence level of the current disparity is obtained, providing a reliable confidence level for focusing. This ensures focusing accuracy and improves the user experience in camera focusing.

[0074] In order to achieve accurate focusing, the determination of parallax is particularly important. The following embodiments of this disclosure provide a detailed explanation of the determination of parallax.

[0075] Figure 2 This is a flowchart illustrating a method for determining the parallax of a left and right view according to an exemplary embodiment. Figure 2 As shown, the current parallax between the left and right views is determined through the following process.

[0076] In step S21a, image features of the original left view image are extracted to obtain the left view feature image, and image features of the original right view image are extracted to obtain the right view feature image.

[0077] After obtaining the original left and right view images, a feature extraction network, such as a convolutional neural network, is used to extract the left view feature image from the original left view image and the right view feature image from the original right view image, in order to save the extracted image features.

[0078] In step S21b, a first disparity map is determined between the left view feature image and the right view feature image, the first disparity map representing the disparity between feature points in the left view feature image and the right view feature image.

[0079] After obtaining the feature images of the left and right views, the disparity between the two feature views is obtained pixel by pixel using a depth estimation neural network. The disparity of each point in the two feature images constitutes the first disparity map. That is, the first disparity map represents the disparity between feature points in the left and right view feature images.

[0080] In one embodiment of this example, such as Figure 3 This is a schematic diagram illustrating a process for determining a disparity map of an original left view and an original right view according to an exemplary embodiment, such as... Figure 3 As shown, the disparity maps of the original left and right views are determined in the following manner.

[0081] Image data for the left and right views are acquired using a phase difference sensor. These two images, a pair of full-pixel dual-core images, can be considered as a small-baseline stereo image pair with the same exposure time and strict correction. The disparity of this image pair corresponds to the degree of image blur, and disparity exists only in the out-of-focus area. After the camera determines the current focus area, based on the focus area, the original left view image and the original right view image are obtained from the preview image. A convolutional neural network is used to extract feature images from the original left and right views, and the feature weights of the two images are shared. A similarity tensor is further calculated using a convolutional neural network. The similarity between the left and right view feature images under different disparity conditions is calculated, and then the disparity is further optimized using a 3D aggregation network. The 3D aggregation network is a 3D convolutional neural network used to downsample the similarity tensor. This outputs the probability of each feature point on the multi-layer feature map under different disparity conditions, ultimately resulting in a first disparity map, i.e., a depth map, the same size as the left and right view feature images.

[0082] In step S21c, the first disparity map is upsampled to the original image size to obtain a second disparity map between the original left view image and the original right view image. The second disparity map represents the disparity between pixels in the feature images of the left view and the right view.

[0083] The first disparity map is upsampled to the size of either the original left or right view image to obtain a second disparity map of the original image size. This yields the PD value for each pixel in the original image, thus creating the second disparity map.

[0084] In step S21d, the current disparity between the left and right views is determined based on the second disparity map.

[0085] Based on the second disparity map, the overall disparity value between the left and right view images is obtained by calculating the disparity value corresponding to each pixel in the second disparity map, for example, by averaging or weighting.

[0086] In this embodiment of the present disclosure, the final disparity value between the two original left view and the original right view can be obtained through the second disparity map, which is used as the disparity value directly for focusing.

[0087] In this embodiment of the present disclosure, the second disparity map is obtained by upsampling the first disparity map, and the first disparity map is obtained by each feature point of the left view feature map and the right view feature map.

[0088] Figure 4 This is a flowchart illustrating a disparity map calculation method according to an exemplary embodiment. For example... Figure 4As shown, the disparity maps of the left view feature map and the right view feature map are completed through the following process.

[0089] In step S31a, for each feature point in the left view feature image and the right view feature image, the first disparity of the target feature point is determined as the target feature point.

[0090] Each feature point in the left and right view feature images is used as a target feature point, and the first disparity value of the target feature point is determined by epipolar matching.

[0091] In one embodiment of this example, the left view is a reference image, and the right view is the image to be matched. Under different depth conditions, for example, there are nine different depths d1, d2, d3, d4, ..., d8, d9, representing the same object point at nine different depths. The projection points of this same object point in the left and right images are the imaging points. A pixel in the reference image is referenced by a projection, and there are nine corresponding projection pixels in the epipolar lines of the image to be matched at different depths. Based on epipolar matching, the sequence of projection points at all these depths forms the search space. In this search space, different projection pixels of the same object point in the reference image and the image to be matched are determined. This determines the display deviation of the same object in the left and right images, i.e., the disparity. For example, if there is a reference point (x1, y1) in the reference image, and through epipolar matching, the corresponding pixel point (x2, y1) is matched in the image to be matched, then the disparity of this point is deltaX = |x1 - x2|, that is, the first disparity of the target feature point between the feature image of the left view and the feature image of the right view is deltaX.

[0092] In step S31b, a first disparity map is obtained based on the first disparity of each target feature point.

[0093] Based on the first disparity deltaX of each target point, the first disparity value of each target point is determined one by one on the corresponding pixels of the left view feature map and the right view feature map, and the first disparity map can be obtained.

[0094] In this embodiment of the disclosure, the method of determining the first disparity deltaX of each target point is applied to the disparity determination of each pixel in the left view feature map and the right view feature map to obtain the first disparity map.

[0095] This disclosure embodiment determines a first disparity map of a left view feature map and a right view feature map, wherein the disparity of each pixel in the disparity map is obtained by each matched target feature point.

[0096] Figure 5 This is a flowchart illustrating a method for calculating the disparity of a target point according to an exemplary embodiment. For example... Figure 5 As shown, the disparity of the target feature points is determined through the following process.

[0097] In step S41a, the similarity of target feature points in the left view feature image and the right view feature image under multiple different disparities is determined, and the probability value of the disparity corresponding to the similarity under multiple different disparities is predicted as the first disparity.

[0098] Based on the disparity of the target feature points determined in the left and right view feature images, the predicted probability value of the point in the feature image pairs of the left and right views obtained at different camera positions is predicted.

[0099] In one embodiment of this invention, different disparity values ​​are obtained at different depths, and these disparity values ​​also correspond to different matching similarities. That is, the horizontal axis represents depth, and the vertical axis represents matching cost, which can represent the matching similarity curve between a pixel in the reference image and nine pixels in the image to be matched at different depths. This can be represented as a pixel-level matching cost, and this pixel-level matching cost constitutes the search space formed by the matching similarities at different depths.

[0100] In one embodiment of this example, a feature pixel in the feature map can form a mapping with a dimension of 1 and a total of 9 pixels. For the entire pixel of the feature map, a similarity tensor with a dimension of W*H*9 can be obtained as the search space, where W is the width of the feature map and H is the height of the feature map.

[0101] In step S41b, the disparity corresponding to the highest probability value is taken as the first disparity of the target feature point.

[0102] Based on the target feature points obtained from the feature images of the left and right views obtained from different predicted camera positions, the probability values ​​of different disparities predicted for these target feature points are used to determine the highest probability value as the disparity of the target feature point, i.e., the first disparity of the target feature point.

[0103] In this embodiment of the disclosure, based on the probability values ​​of the target feature point in different disparity predictions, the disparity corresponding to the highest probability value is determined as the disparity value of the target feature point. This probability value is obtained after conversion through similarity matching.

[0104] Figure 6 This is a flowchart illustrating a method for determining target point similarity matching according to an exemplary embodiment. Figure 6 As shown, the similarity of target feature points under different disparities is determined through the following process.

[0105] In step S51a, one of the left view feature image and the right view feature image is used as a reference image, and the other is used as the image to be matched.

[0106] Choose one of the left-view feature image and the right-view feature image as the reference image for target point similarity matching, and the other as the image to be matched for target point similarity matching.

[0107] In step S51b, the target feature points in the reference image are mapped to the image to be matched under multiple different disparities, resulting in multiple target feature points to be matched corresponding to multiple different disparities.

[0108] In the reference image, a point is identified as the target feature point. Based on epipolar matching, similarity matching is performed on the pixel sequence in the same row as the target feature point in the image to be matched, which can yield multiple target feature points with different disparities.

[0109] In step S51c, the similarity between the target feature point and multiple target feature points to be matched is used as the similarity of the target feature point under multiple different disparities.

[0110] The similarity between a target feature point and multiple target feature points to be matched is evaluated using a similarity estimation algorithm. For example, SAD can be used to calculate the similarity between a 3x3 region around the target feature point and a 3x3 region around the target feature point to be matched. The similarity between the two can be calculated for each target feature point to be matched and used as the similarity of the target feature point under multiple different disparities.

[0111] In this embodiment of the disclosure, by referring to the target feature points in the figure and the target feature points to be matched in the figure to be matched, similarity matching is performed in the search space to be matched determined by epipolar matching, thereby determining the similarity of the target feature points under multiple different disparities.

[0112] This disclosure describes an embodiment that determines the similarity of target feature points under different disparities, and the disparity determination is based on feature images.

[0113] Figure 7 This is a flowchart illustrating a method for feature extraction from a left-view original image and a right-view original image according to an exemplary embodiment. Figure 7 As shown, the feature images of the left view and the right view were obtained through the following process.

[0114] In step S61a, the original left view image and the original right view image are respectively input into the first network model and weight sharing is performed to extract the image features of the original left view image and the image features of the original right view image.

[0115] The first network model is a feature extraction network, such as a Convolutional Neural Network (CNN), a Visual Geometry Group (VGG) CNN, or a Residual Network (ResNet). When extracting features from the original left and right view images based on this feature extraction network, weight sharing is used to extract the image features of both the original left and right view images.

[0116] In step S61b, the left view feature image and the right view feature image are obtained.

[0117] The image features of the original image of the left view are used as the feature image of the left view, and the image features of the original image of the right view are used as the feature image of the right view.

[0118] In this embodiment of the disclosure, feature extraction is performed on the original left and right view images by a feature extraction neural network to obtain the feature images of the left and right views. This provides rich and reliable feature image data for disparity prediction. Compared with the traditional method of directly relying on the original images for disparity calculation, this embodiment of the disclosure can provide rich image features, which helps to stabilize the feature extraction in disparity calculation.

[0119] In this embodiment of the disclosure, a data source for stable disparity calculation is obtained based on the left-view feature image and the right-view feature image extracted by the feature extraction neural network, and disparity prediction can be performed based on this data source.

[0120] Figure 8 This is a flowchart illustrating a method for predicting disparity using a depth estimation neural network according to an exemplary embodiment. Figure 8 As shown, disparity prediction was accomplished through the following process.

[0121] In step S71a, the target feature points are input into the second network model. The input of the second network model is the feature points, and the output is the disparity.

[0122] After obtaining the left and right view feature images through the feature extraction neural network, the target feature points in the two images are used as the input of the input layer of the second network model. The second network model is a depth estimation neural network, which is used to predict which type of disparity the target feature points correspond to in different disparity categories.

[0123] In step S71b, the first disparity is obtained based on the output of the second network model.

[0124] By using a deep estimation neural network model, the disparity corresponding to the target feature point is predicted as the first disparity.

[0125] In this embodiment of the disclosure, a second network model is used to determine the disparity values ​​corresponding to target feature points in the left view feature image and the right view feature image. This second network model is trained using the following method.

[0126] Figure 9 This is a flowchart illustrating a training method for a second network model according to an exemplary embodiment. Figure 9 As shown, the training of the second network model is completed through the following process.

[0127] In step S81a, the original image sequence is obtained.

[0128] During the process of the image acquisition device's lens being pushed linearly from its nearest point to its farthest point, or from its farthest point to its nearest point, by the motor, the image acquisition device continuously acquires images to obtain the original image sequence. The motor's pushing step size is the minimum linearly movable step size.

[0129] In step S81b, the clearest image in the original image sequence is determined, the position corresponding to the clearest image is obtained, and the disparity map of each image corresponding to the position is determined according to the ratio between position and disparity.

[0130] After obtaining the original image sequence, the sharpest image is determined from it. This can be done using sharpness evaluation algorithms, such as the Tenengrad method, the Laplacian gradient method, and the variance method. This identifies the sharpest image in the sequence, and from this, the shooting position (depth) corresponding to that sharpest image is determined. Based on the ratio of position to parallax, a disparity map for each image at different shooting positions can be determined.

[0131] In step S81c, a second network model is trained based on the image sequence and the corresponding disparity map.

[0132] Based on the disparity map corresponding to the image taken at the determined current depth, disparity maps of all depths are used as training labels, and feature images obtained by feature extraction from images taken at all depths are used as training data. This training data and training labels constitute the training sample dataset. The second network model is then trained using this dataset. The loss function can employ Mean Squared Error (MSE) or Smooth L1 to evaluate the distance between the predicted disparity and the true disparity.

[0133] In this embodiment of the disclosure, a training sample dataset for a second network model is generated and trained. When this model is used to predict disparity, the confidence level of the current disparity can be determined by the predicted disparity value and the corresponding probability value.

[0134] Figure 10 This is a flowchart illustrating a method for determining the current disparity confidence level according to an exemplary embodiment. Figure 10 As shown, the confidence level of the current disparity was determined through the following process.

[0135] In step S91a, the disparity probability distribution curve is determined based on the disparity probability distribution.

[0136] Using the second network model described above, the probability value corresponding to the current disparity being classified into different disparities is predicted, thereby determining the disparity probability distribution curve.

[0137] In step S91b, the first area is obtained by using the disparity probability distribution curve, with the probability value corresponding to the current disparity value as the high value and the difference between the maximum and minimum values ​​of different disparities as the wide value.

[0138] Based on the probability values ​​corresponding to different disparities in the disparity probability distribution curve, the probability value corresponding to the current disparity value is set as high, that is, the highest probability value is determined as high, and the difference between the maximum and minimum values ​​of the depths corresponding to all disparities is taken as the width. The area value of the rectangular region formed by the calculation is taken as the first area S1.

[0139] In step S91c, a second area is determined between the disparity probability distribution curve and the disparity axis between the maximum and minimum values ​​of different disparities.

[0140] The area formed by the parallax probability distribution curve and the parallax axis across all parallaxes is the second area S2.

[0141] In step S91d, the confidence level of the current disparity is determined based on the first area and the second area.

[0142] Based on the first area S1 and the second area S2, the confidence level of the current parallax is Confidence = S1 / S2.

[0143] In one embodiment of this disclosure, such as Figure 11 The diagram shows a disparity probability distribution curve. The probability value corresponding to the current disparity value is taken as the height h = 0.47, and the difference between the maximum and minimum values ​​of different disparities is taken as the width w = 580 - 20 = 560. This yields the area S1 = h * w = 0.47 * 560 = 263.2, which is used as the first area S1 = 263.2. Based on the weighted sum of each disparity and its corresponding depth, i.e., through... We can obtain S2, where Dmax represents the maximum disparity, k represents different disparities, and pk represents the probability under that disparity. In this example, S2 = 150, so the confidence level of this disparity is Confidence1 = S1 / S2 = 263.2 / 150 = 1.75.

[0144] In one embodiment of this disclosure, such as Figure 12 The diagram shows a disparity probability distribution curve. In this example, the first area S1 = 0.83 * (595 - 5) = 489.7, and the second area S2 = 93. The confidence level of this disparity is Confidence2 = S1 / S = 489.7 / 93 = 5.27.

[0145] Based on the comparison of the two implementation methods above, Confidence1 is less than Confidence2. Obviously, the probability value of the current disparity in Confidence2 is higher, and the probability values ​​of the disparities adjacent to the current disparity are also higher. Therefore, the confidence level of Confidence2 is more reliable than that of Confidence1.

[0146] In one embodiment of this disclosure, such as Figure 13 The comparison shows the calculation results of the traditional parallax calculation method and the method of this disclosure under both bright and dark environments. Figure 13 As shown, the horizontal axis represents the lens position, and the vertical axis represents the calculated PD value. 1000lux_TE42_pd represents the curve of PD value and lens position calculated using a traditional algorithm in an environment with a brightness of 1000 lux; 1000lux_TE42_mlpd represents the curve of PD value and lens position calculated using the technical solution of this disclosure in an environment with a brightness of 1000 lux; 1lux_TE42_pd represents the curve of PD value and lens position calculated using a traditional algorithm in an environment with a brightness of 1 lux; 1lux_TE42_mlpd represents the curve of PD value and lens position calculated using the technical solution of this disclosure in an environment with a brightness of 1 lux. That is, in a low-light environment with an illumination of 1 lux, the relationship curve between the lens position cur_lens_pos and parallax pd obtained by the traditional PD algorithm fluctuates greatly and does not reflect the relationship between the two. However, using the technical solution proposed in this disclosure, a near-linear relationship between cur_lens_pos and pd is obtained, demonstrating the excellent focusing performance of the technical solution of this disclosure in low-light environments. In backlit conditions with an illumination of 1000 lux, traditional PD algorithms fail to reflect the linear relationship between cur_lens_pos and pd at many lens positions. However, the technical solution disclosed in this paper demonstrates a very accurate linear relationship between cur_lens_pos and pd in backlit conditions, achieving excellent performance.

[0147] In one embodiment of this example, for instance, the confidence threshold is set to 4. When the confidence level is greater than 4, it is determined that the predicted disparity accuracy is high, and the lens can be adjusted to the focus point by moving the motor in specified step units according to the currently predicted disparity value. If the confidence level is less than or equal to 4, it is determined that the predicted disparity accuracy is low and insufficient to guarantee that the currently predicted disparity value is accurate, and the adjustment of the lens position by moving the motor is abandoned.

[0148] Figure 14 This is a flowchart illustrating a focusing method according to an exemplary embodiment. For example... Figure 14 As shown, the camera begins taking pictures, exposure is complete, and left and right view preview images are obtained. Based on the focus area, the original left and right view images are obtained. Based on the original left and right view images, image features are extracted using a convolutional neural network, and the feature images are input into the decision network to predict the type of pixel offset, i.e., the disparity category. The pixel offsets corresponding to the largest predicted values ​​in the pixel offset category are taken as the disparity PD values. Confidence is calculated based on the curve of the multi-category pixel offset values ​​and their corresponding probability values. The confidence threshold is used to determine whether the disparity PD is reliable; if reliable, the motor is moved to focus based on the PD value.

[0149] Figure 15 This is a flowchart illustrating a focusing method according to an exemplary embodiment. For example... Figure 15As shown, the camera is opened, exposure is completed, and preview images of the left and right views are obtained. Based on the focus area, valid original images of the left and right views are obtained. The focus area can be determined through methods such as center focus, touch focus, face focus, and eye focus. After obtaining the original images of the left and right views, disparity estimation can be performed. Disparity estimation involves extracting image features from the original left and right views using a convolutional neural network, with weight sharing during feature extraction. After feature extraction, feature images of the left and right views are obtained. Based on these feature images, a similarity tensor is calculated using epipolar matching to represent the similarity of each feature point in the two images. Then, a 3D aggregation network is used to downsample the 3D tensor of the similarity tensor through 3D convolution. A multi-classification network converts the similarity into probability values ​​for different disparity classifications. The disparity corresponding to the highest probability value is determined as the accurate disparity value at that point, thus obtaining a disparity mapping between the feature images of the left and right views. Each pixel in the disparity map contains the disparity value at that point, and the size of the disparity map is the same as that of the left and right view feature maps. Then, the upsampled value of this disparity map is made the same size as the original left and right view images. This upsampling can be performed using deconvolution. Based on the above processing, the disparity maps corresponding to the original left and right view images can be determined.

[0150] Based on the focus area, the region of interest (ROI) in the disparity map is determined, and the disparity value and corresponding confidence value within this region are calculated. The ROI can be determined using a multi-window strategy, such as... Figure 16 This is a multi-window strategy illustrated according to an exemplary embodiment. For example... Figure 16 This is a schematic diagram illustrating a multi-window strategy according to an exemplary embodiment. For example... Figure 16 As shown, the ROI can be determined by the central area of ​​the preview screen, or by detecting areas such as the face area or the eye area.

[0151] The parallax value with the highest probability among different parallax values ​​is selected as the parallax value for the adjustable motor, and the confidence level of this parallax value is determined. If the confidence level meets the confidence threshold, the current reliable parallax PD value can be converted into the motor's drive distance, and the motor can then be translated.

[0152] This completes the focusing process.

[0153] Based on the same concept, this disclosure also provides a focusing device 100.

[0154] It is understood that the focusing device 100 provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0155] Figure 17 This is a block diagram illustrating a focusing device 100 according to an exemplary embodiment. (Refer to...) Figure 17 The device includes an acquisition unit 101, a parallax determination unit 102, a confidence determination unit 103, and a focusing unit 104.

[0156] The acquisition unit 101 is configured to focus in response to the preview screen acquired by the image acquisition device, and to obtain the original left view image and the original right view image corresponding to the focus area based on the focus area in the preview screen.

[0157] The disparity determination unit 102 is configured to determine the current disparity between the original left view image and the original right view image, and to determine the disparity probability distribution corresponding to the current disparity. The disparity probability distribution characterizes the correspondence between multiple different disparities and probability values. The multiple different disparities include the current disparity and multiple disparities with different disparity offsets. The probability values ​​characterize the degree of matching between the different disparities and the current disparity.

[0158] The confidence determination unit 103 is configured to determine the confidence level of the current disparity based on the disparity probability distribution.

[0159] The focusing unit 104 is configured to focus based on the parallax in response to the confidence level meeting a confidence threshold.

[0160] In one embodiment, the disparity determination unit 102 determines the current disparity between the original left view image and the original right view image in the following manner:

[0161] Extract the image features of the original left view image to obtain the left view feature image, and extract the image features of the original right view image to obtain the right view feature image;

[0162] A first disparity map is determined between the left view feature image and the right view feature image, the first disparity map representing the disparity between feature points in the left view feature image and the right view feature image;

[0163] The first disparity map is upsampled to the original image size to obtain a second disparity map between the original left view image and the original right view image. The second disparity map represents the disparity between pixels in the feature image of the left view and the feature image of the right view.

[0164] Based on the second disparity map, the current disparity between the left view and the right view is determined.

[0165] In one embodiment, the disparity determination unit 102 determines a first disparity map between the left view feature image and the right view feature image in the following manner:

[0166] For each feature point in the left view feature image and the right view feature image, respectively, a target feature point is selected, and the first disparity of the target feature point is determined.

[0167] Based on the first disparity of each of the target feature points, a first disparity map is obtained;

[0168] Wherein, the first disparity for determining the target feature point includes:

[0169] Determine the similarity of target feature points in the left view feature image and the right view feature image under multiple different disparities, and predict the probability value of the disparity corresponding to the similarity under multiple different disparities as the first disparity;

[0170] The disparity corresponding to the highest probability value is taken as the first disparity of the target feature point.

[0171] In one embodiment, the disparity determination unit 102 determines the similarity of target feature points in the left view feature image and the right view feature image under multiple different disparities in the following manner:

[0172] One of the left view feature image and the right view feature image is used as a reference image, and the other is used as the image to be matched;

[0173] The target feature points in the reference image are mapped to the image to be matched under multiple different disparities to obtain multiple target feature points to be matched corresponding to the multiple different disparities;

[0174] The similarity between the target feature point and the plurality of target feature points to be matched is taken as the similarity of the target feature point under multiple different disparities.

[0175] In one embodiment, the disparity determination unit 102 is further configured to determine the disparity probability distribution based on the first disparity and the probability value of the predicted disparity corresponding to the similarity under the plurality of different disparities as the first disparity.

[0176] In one embodiment, the disparity determination unit 102 extracts image features from the original left view image to obtain a left view feature image, and extracts image features from the original right view image to obtain a right view feature image:

[0177] The original left view image and the original right view image are respectively input into the first network model and weights are shared to extract the image features of the original left view image and the original right view image, so as to obtain the left view feature image and the right view feature image.

[0178] The first network model takes an image as input and outputs image features.

[0179] In one embodiment, the disparity determination unit 102 determines the first disparity of the target feature point in the following manner:

[0180] The target feature points are input into the second network model, where the input to the second network model is the feature points and the output is the disparity.

[0181] The first disparity is obtained based on the output of the second network model.

[0182] In one embodiment, the disparity determination unit 102 trains the second network model in the following manner:

[0183] Obtain the original image sequence;

[0184] The sharpest image is determined from the original image sequence, and the position corresponding to the sharpest image is obtained. Based on the ratio between the position and the disparity, the disparity map of each image corresponding to the position is determined.

[0185] The second network model is trained based on the image sequence and the corresponding disparity map.

[0186] In one embodiment, the confidence determination unit 103 determines the confidence level of the current disparity based on the disparity probability distribution in the following manner:

[0187] Based on the disparity probability distribution, determine the disparity probability distribution curve;

[0188] Using the disparity probability distribution curve, the first area is obtained by taking the probability value corresponding to the current disparity value as high and the difference between the maximum and minimum values ​​of different disparities as wide.

[0189] Determine the second area formed by the disparity probability distribution curve and the disparity axis between the maximum and minimum values ​​of different disparities.

[0190] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0191] Figure 18 This is a block diagram illustrating a focusing device 200 according to an exemplary embodiment. For example, device 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0192] Reference Figure 18 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0193] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0194] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0195] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0196] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0197] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0198] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0199] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0200] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0201] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0202] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0203] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0204] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0205] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0206] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0207] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0208] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0209] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A focusing method, characterized in that, Applied to terminals, including: In response to the preview screen captured by the image acquisition device in the terminal being triggered to focus, the original left view image and the original right view image corresponding to the focus area are obtained based on the focus area in the preview screen. The current disparity between the original left view image and the original right view image is determined, and the disparity probability distribution corresponding to the current disparity is determined. The disparity probability distribution represents the correspondence between multiple different disparities and probability values. The multiple different disparities include the current disparity and multiple disparities with different disparity offsets from the current disparity. The probability values ​​represent the degree of matching between the different disparities and the current disparity. Based on the disparity probability distribution, determine the confidence level of the current disparity; In response to the confidence level meeting the confidence threshold, focusing is performed based on the parallax.

2. The method according to claim 1, characterized in that, Determining the current disparity between the original left view image and the original right view image includes: Extract the image features of the original left view image to obtain the left view feature image, and extract the image features of the original right view image to obtain the right view feature image; A first disparity map is determined between the left view feature image and the right view feature image, the first disparity map representing the disparity between feature points in the left view feature image and the right view feature image; The first disparity map is upsampled to the original image size to obtain a second disparity map between the original left view image and the original right view image. The second disparity map represents the disparity between pixels in the feature image of the left view and the feature image of the right view. Based on the second disparity map, the current disparity between the left view and the right view is determined.

3. The method according to claim 2, characterized in that, Determining the first disparity map between the left view feature image and the right view feature image includes: For each feature point in the left view feature image and the right view feature image, respectively, a target feature point is selected, and the first disparity of the target feature point is determined. Based on the first disparity of each of the target feature points, a first disparity map is obtained; Wherein, the first disparity for determining the target feature point includes: Determine the similarity of target feature points in the left view feature image and the right view feature image under multiple different disparities, and predict the probability value of the disparity corresponding to the similarity under multiple different disparities as the first disparity; The disparity corresponding to the highest probability value is taken as the first disparity of the target feature point.

4. The method according to claim 3, characterized in that Determining the similarity of target feature points in the left view feature image and the right view feature image under multiple different disparities includes: One of the left view feature image and the right view feature image is used as a reference image, and the other is used as the image to be matched; The target feature points in the reference image are mapped to the image to be matched under multiple different disparities to obtain multiple target feature points to be matched corresponding to the multiple different disparities; The similarity between the target feature point and the plurality of target feature points to be matched is taken as the similarity of the target feature point under multiple different disparities.

5. The method according to claim 3 or 4, characterized in that, The disparity probability distribution is determined based on the first disparity and the probability value of the disparity corresponding to the similarity under the multiple different disparities as the first disparity.

6. The method according to claim 2, characterized in that, The process of extracting image features from the original left view image to obtain a left view feature image, and extracting image features from the original right view image to obtain a right view feature image, includes: The original left view image and the original right view image are respectively input into the first network model and weights are shared to extract the image features of the original left view image and the original right view image, so as to obtain the left view feature image and the right view feature image. The first network model takes an image as input and outputs image features.

7. The method according to claim 3, characterized in that, The first disparity for determining the target feature point includes: The target feature points are input into the second network model, where the input to the second network model is the feature points and the output is the disparity. The first disparity is obtained based on the output of the second network model.

8. The method according to claim 7, characterized in that, The second network model was trained in the following manner: Obtain the original image sequence; The sharpest image is determined from the original image sequence, and the position corresponding to the sharpest image is obtained. Based on the ratio between the position and the disparity, the disparity map of each image corresponding to the position is determined. The second network model is trained based on the image sequence and the corresponding disparity map.

9. The method according to claim 1, characterized in that, The step of determining the confidence level of the current disparity based on the disparity probability distribution includes: Based on the disparity probability distribution, determine the disparity probability distribution curve; Using the disparity probability distribution curve, the first area is obtained by taking the probability value corresponding to the current disparity value as high and the difference between the maximum and minimum values ​​of different disparities as wide. Determine the second area formed by the disparity probability distribution curve and the disparity axis between the maximum and minimum values ​​of different disparities; The confidence level of the current parallax is determined based on the first area and the second area.

10. A focusing device, characterized in that, The focusing device includes: The acquisition unit is used to focus in response to the preview screen acquired by the image acquisition device being triggered, and to obtain the original left view image and the original right view image corresponding to the focus area based on the focus area in the preview screen. A disparity determination unit is used to determine the current disparity between the original left view image and the original right view image, and to determine the disparity probability distribution corresponding to the current disparity. The disparity probability distribution characterizes the correspondence between multiple different disparities and probability values. The multiple different disparities include the current disparity and multiple disparities with different disparity offsets from the current disparity. The probability values ​​characterize the degree of matching between the different disparities and the current disparity. A confidence determination unit is used to determine the confidence level of the current disparity based on the disparity probability distribution. A focusing unit is configured to focus based on the parallax in response to the confidence level meeting a confidence threshold.

11. A focusing device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the focusing method according to any one of claims 1 to 9.

12. A storage medium, characterized in that, The storage medium stores instructions that, when executed by the terminal's processor, enable the terminal to perform the focusing method as described in any one of claims 1 to 9.

Images