A PDAF confidence calculation method based on deep learning

Abstract

The present invention provides a PDAF confidence calculation method based on deep learning, comprising: collecting image data and performing data preprocessing, and simultaneously performing PDAF classification network design; training PDAF classification network parameters to obtain a PDAF classification network model; inputting a PD image into the PDAF classification network model, outputting a probability distribution of a Defocus value; and outputting a Defocus value and a confidence according to the probability distribution of the Defocus value. The present invention can output not only a Defocus value but also a corresponding confidence according to the probability distribution output by the PDAF classification network model by training the PDAF classification network parameters, thereby effectively avoiding focusing errors caused by CNN PDAF prediction errors.

Description

Technical Field

[0001] The present invention relates to the field of camera imaging technology, and in particular to a PDAF confidence calculation method based on deep learning. Background Art

[0002] The traditional focusing method is contrast focusing, which uses the contrast of the image as a reference standard. The point with the highest contrast is selected as the focus point during the focusing process. The motor drives the lens to move from the lowest point to the highest point, and records the image contrast at each position. After the lens has completed the entire process, it moves the lens to the position with the highest contrast to complete contrast focusing.

[0003] The basic principle of PDAF (Phase Detection Auto Focus) is that the part of the mobile phone module that senses the image is a photosensitive chip, and each pixel is sensing the image. If two symmetrical pixels separated by a certain distance are covered with the left and right halves of the pixels respectively, it is equivalent to the left and right eyes of a person. According to the different angles at which the two eyes see the object, it can be calculated whether the focus is accurate.

[0004] Compared with contrast focus, PDAF does not require the motor to search the entire process. Instead, it can determine the distance the motor needs to reach the clear point by judging the phase difference, and push the lens to that clear point at once, thereby greatly improving the focusing speed.

[0005] However, since the task target of PDAF is the Defocus value (DAC value required to move to the focus, DAC is the position where the motor moves), CNN (Convolutional Neural Network) PDAF usually uses a regression model for modeling. However, the CNN PDAF regression model cannot output and use confidence to ensure the stability of the PDAF application like the traditional method, avoid using wrong PDAF prediction results, and thus affect the application of the CNN PDAF algorithm.

[0006] In 2019, Nubia Technology Co., Ltd. submitted a patent application with the patent name "A confidence optimization method, device and computer-readable storage medium", and the patent number is CN110233962A. The confidence optimization method provided by the patent has higher confidence calculation accuracy, making the confidence threshold more reasonable and accurate, but this method is limited to some difficult scenarios and the calculation method cannot be transferred to deep learning, and this method requires noise compensation parameters to guide the calculation of confidence. Summary of the invention

[0007] The object of the present invention is to provide a PDAF confidence calculation method based on deep learning, which can not only output a Defocus value, but also output a corresponding confidence, and can effectively avoid focusing errors caused by CNN PDAF prediction errors.

[0008] In order to solve the above technical problems, the present invention provides a PDAF confidence calculation method based on deep learning, comprising:

[0009] Collect image data and perform data preprocessing, and design PDAF classification network;

[0010] Train PDAF classification network parameters to obtain a PDAF classification network model;

[0011] Inputting the PD image into the PDAF classification network model, and outputting the probability distribution of Defocus values; and

[0012] The Defocus value and the confidence level are output according to the probability distribution of the Defocus value.

[0013] Optionally, the scenes for collecting image data include normal scenes, high-frequency scenes, moiré scenes and / or dark light scenes.

[0014] Optionally, the collected image data includes a PD image, and the PD image is divided into 6*9 image blocks of 100*100*2 pixels, where *2 indicates that the training image PD data is split into left and right images.

[0015] Optionally, the method for data preprocessing includes generating a Defocus label and a confidence label.

[0016] Optionally, methods for generating a Defocus tag include:

[0017] Calculate the contrast of the acquired image;

[0018] Find N points near the maximum gradient value and calculate the focus point position, where N is a positive integer; and

[0019] A Defocus tag is generated according to the focus point position.

[0020] Optionally, the calculation formula for the contrast of the image is:

[0021]

[0022] G x (x,y)=2I(x,y)-I(x-1,y)-I(x+1,y)

[0023] G y(x,y)=2I(x,y)-I(x,y-1)-I(x,y+1)

[0024] Here, x and y refer to the position of the pixel in the image.

[0025] Optional, Defocus tag:

[0026] Label=DAC current -DAC focus

[0027] Among them, DAC current Refers to the DAC value of the existing acquired image. DAC focus It refers to the DAC value corresponding to the focus point position, and DAC refers to the position where the motor moves.

[0028] Optionally, methods for generating confidence labels include:

[0029] Determine the Label interval range and Confidence_Label value, where Confidence_Label = Label + 499;

[0030] Determine, according to the Confidence_Label value, the interval in which the Label value corresponding to each of the images falls and the probability of falling within the interval; and

[0031] Generate confidence labels.

[0032] Optionally, the method for designing a PDAF classification network includes:

[0033] Establish a CNN network model;

[0034] Extract feature information of CNN network;

[0035] Connect to the fully connected layer and the normalized exponential function.

[0036] Optionally, according to the characteristics of the image data, the first layer is set to a two-channel input with a size of height*width*2.

[0037] Optionally, extracting feature information of the CNN network includes: extracting feature layers of N different depths in the network, respectively passing through a 1*1 convolutional layer and a fully connected layer to obtain an M-dimensional feature vector, and adding offsets Dx and Dy from the center point to form N*M+2 dimensional features.

[0038] Optionally, a method for training PDAF classification network parameters includes:

[0039] Use data augmentation methods to further expand the data set and improve the model's performance;

[0040] Use uniform distribution initialization;

[0041] The KL-divergence loss function is used to measure the approximate complex target distribution.

[0042] Optionally, approximate the complexity of the target distribution using the following formula:

[0043]

[0044] Among them, p is the original probability distribution and q is the approximate probability distribution.

[0045] Optionally, the method for training PDAF classification network parameters also includes: selecting the model with the best performance in actual testing based on the two models with the smallest loss function and the highest classification accuracy in the validation set.

[0046] Optionally, the method of outputting the Defocus value and the confidence level according to the probability distribution includes:

[0047] Calculating the confidence level according to the probability distribution and expected distribution of the output Defocus value;

[0048] According to the probability distribution of the output Defocus value, find the maximum probability interval and calculate the predicted Defocus value.

[0049] In summary, in the PDAF confidence calculation method based on deep learning provided by the present invention, image data is first collected and data preprocessing is performed, and a PDAF classification network design is performed at the same time, and then the PDAF classification network parameters are trained to obtain a PDAF classification network model, and then the PD image is input to the PDAF classification network model, and the probability distribution of the Defocus value is output, and the Defocus value and the confidence are output according to the probability distribution of the Defocus value. The present invention can not only output the Defocus value but also the corresponding confidence by training the PDAF classification network parameters according to the probability distribution output by the PDAF classification network model, which can effectively avoid the focus error caused by the CNN PDAF prediction error. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0051] Figure 1 It is a flowchart of a PDAF confidence calculation method based on deep learning provided by one embodiment of the present invention.

[0052] Figure 2It is a flowchart of a PDAF confidence calculation method based on deep learning provided by an embodiment of the present invention.

[0053] Figure 3 FIG. 4 is a schematic diagram of the relationship between contrast and DAC provided by an embodiment of the present invention.

[0054] Figure 4 Yes Figure 3 The curve diagram after fitting the data within the dashed circle.

[0055] Figure 5 It is a flowchart of a PDAF classification network design provided by an embodiment of the present invention.

[0056] Figure 6a~6d It is a schematic diagram of the relationship between Defocus, confidence and DAC provided by an embodiment of the present invention.

[0057] Figure 7 It is a flowchart of calculating confidence using a PDAF confidence calculation method based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the drawings is often a part of the actual structure. In particular, the emphasis of each drawing is different, and sometimes different scales are used.

[0059] As used in the present invention, the singular forms "a", "an" and "the" include plural objects, the term "or" is generally used in a sense including "and / or", the term "several" is generally used in a sense including "at least one", and the term "at least two" is generally used in a sense including "two or more". In addition, the terms "first", "second" and "third" are used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" and "third" may explicitly or implicitly include one or at least two of such features. The term "proximal end" is usually the end close to the operator, and the term "distal end" is usually the end close to the patient. "One end" and "the other end" as well as "proximal end" and "distal end" usually refer to two corresponding parts, which not only include endpoints. The terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integrated connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements.

[0060] In addition, as used in the present invention, an element disposed on another element generally only indicates that there is a connection, coupling, cooperation or transmission relationship between the two elements, and the connection, coupling, cooperation or transmission between the two elements may be direct or indirect through an intermediate element, and cannot be understood as indicating or implying the spatial position relationship between the two elements, that is, one element may be in any orientation such as inside, outside, above, below or on one side of another element, unless otherwise clearly indicated in the content. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] Figure 1 is a flow chart of a PDAF confidence calculation method based on deep learning provided by an embodiment of the present invention. Figure 1 As shown, the present invention provides a PDAF confidence calculation method based on deep learning, comprising:

[0062] S1: collect image data and perform data preprocessing, and design PDAF classification network;

[0063] S2: training PDAF classification network parameters to obtain a PDAF classification network model;

[0064] S3: Input the PD image to the PDAF classification network model, and output the probability distribution of Defocus values; and

[0065] S4: Outputting a Defocus value and a confidence level according to the probability distribution of the Defocus value.

[0066] Figure 2 FIG. 1 is a flowchart of a PDAF confidence calculation method based on deep learning provided by an embodiment of the present invention. Figure 1 and Figure 2 A PDAF confidence calculation method based on deep learning provided by an embodiment of the present invention is described in detail.

[0067] In step S1, image data is collected and data preprocessing is performed, and a PDAF classification network design is performed at the same time. First, image data is collected to obtain a data set. In this embodiment, the scene for collecting image data may include a normal scene, a high-frequency scene, a moiré scene, and / or a dark light scene, but is not limited thereto. The collected image data includes a PD (phase difference) image. Exemplarily, the PD image is divided into 6*9 image blocks of 100*100*2 pixels, where *2 indicates that the training image PD data is split into left and right images.

[0068] Next, the collected data is preprocessed. In this embodiment, the method for preprocessing the data includes generating a Defocus (DAC value required to move to the focus, DAC is the position of the motor movement) label and a confidence label.

[0069] Exemplarily, the method for generating a Defocus tag includes: calculating the contrast of the captured image; finding N points near the maximum gradient value and calculating the focus point position, where N is a positive integer; and generating a Defocus tag according to the focus point position.

[0070] Specifically, first, the contrast of the image is calculated, and the calculation formula of the contrast is:

[0071]

[0072] G x (x,y)=2I(x,y)-I(x-1,y)-I(x+1,y)

[0073] G y (x,y)=2I(x,y)-I(x,y-1)-I(x,y+1)

[0074] Here, x and y refer to the position of the pixel in the image.

[0075] Next, find N points near the maximum gradient value and calculate the focus position. Figure 3 FIG. 1 is a schematic diagram of the relationship between contrast and DAC provided by an embodiment of the present invention, that is, a schematic diagram of a contrast curve. Figure 3As shown in the figure, the horizontal axis is DAC (the position of the motor movement), the vertical axis is contrast (Contrast), and the area where the dotted circle is located contains N points near the maximum gradient value. Next, the data in the dotted circle is fitted to obtain a quadratic curve graph, as shown in Figure 4 As shown, it is Figure 3 The curve diagram after fitting the data in the dotted circle, from which the focus point position can be obtained, such as Figure 4 Shown by the dashed line.

[0076] Finally, a Defocus tag is generated according to the focus point position. The Defocus tag is:

[0077] Label=DAC current -DAC focus

[0078] Among them, DAC current Refers to the DAC value of the existing acquired image. DAC focus Refers to the DAC value corresponding to the focus point position.

[0079] Exemplarily, the method for generating a confidence label includes: first, determining a Label interval range and a Confidence_Label (trust label) value, wherein Confidence_Label=Label+499, and the Label interval range is, for example, [0, 1000].

[0080] Next, the interval in which the Label value corresponding to each of the images falls and the probability of falling within the interval are determined according to the Confidence_Label value. In this embodiment, two points are used to express the interval in which the Label value falls and the probability of falling within the interval, for example:

[0081]

[0082] In the above formula, Label refers to the Confidence_Label value, and K refers to the number of intervals, that is, there are K intervals. In this embodiment, Confidence_Label=549, K=20 is used as an example for explanation. When the Label interval range is [0,1000], K=20, then the length of each interval is 50. The above calculation results 27 and 28 indicate that the Label values ​​corresponding to the image are in the 27 interval and the 28 interval.

[0083] Then, according to the above formulas of z1 and z2, the following formula can be obtained:

[0084]

[0085] In the above formula, Label refers to the Confidence_Label value, and K refers to the number of intervals, that is, there are K intervals. In this embodiment, Confidence_Label = 549, K = 20 is used as an example for explanation. 0.55 means that the probability that the Label value corresponding to the image falls within the 27 interval is 0.55, and 0.45 means that the probability that the Label value corresponding to the image falls within the 28 interval is 0.55.

[0086] Finally, confidence labels are generated, 50 → [0, …, 0.55, 0.45, 0, …, 0].

[0087] Figure 5 This is a flowchart of the PDAF classification network design provided by an embodiment of the present invention, please refer to Figure 5 As shown, the method for designing a PDAF classification network includes: first establishing a model, that is, establishing a CNN (Convolutional Neural Network, feedforward neural network) network model; then extracting feature information, that is, extracting feature information of the CNN network; followed by fc / softmax, where fc refers to a fully connected layer and softmax refers to a normalized exponential function. That is, the method for designing a PDAF classification network includes: establishing a CNN network model; extracting feature information of the CNN network; and connecting a fully connected layer after the feature layer, and finally connecting a normalized exponential function.

[0088] In the step of inputting the collected image data, according to the characteristics of the image data (ie, the characteristics of the PD data), the first layer is set to a two-channel input, namely, left and right images, with a size of height*width*2.

[0089] In the step of extracting feature information of the CNN network, N feature layers of different depths in the network are extracted, and M-dimensional feature vectors are obtained through 1*1 convolutional layers and fully connected layers respectively. The offsets Dx and Dy from the center point are added to form N*M+2 dimensional features.

[0090] In step S1, the steps of collecting image data and performing data preprocessing and the step of designing a PDAF classification network can be performed simultaneously.

[0091] In step S2, the PDAF classification network parameters are trained to obtain a PDAF classification network model.

[0092] In this embodiment, the method for training the PDAF classification network parameters includes: using a data enhancement method to further expand the data set to improve the effect of the model; using uniform distribution initialization; using a KL-divergence loss function to measure the approximate complex target distribution.

[0093] The formula for measuring the approximate complexity of the target distribution is:

[0094]

[0095] Among them, p is the original probability distribution and q is the approximate probability distribution.

[0096] In this embodiment, the optimizer used is Adam (Adaptive Moment Estimation). After setting the network hyperparameters, the network model is trained using Pyotrch, and the model with the best performance is selected in the actual test based on the two models with the smallest loss function and the highest classification accuracy in the validation set.

[0097] In step S3, the PD image is input into the PDAF classification network model, and the probability distribution of the Defocus value is output.

[0098] The PD image is input into the PDAF classification network model, and the probability distribution of Defocus values ​​is output.

[0099] In step S4, the Defocus value and the confidence level are output according to the probability distribution of the Defocus value.

[0100] Specifically, the confidence is calculated according to the probability distribution of the output Defocus value and the expected distribution; according to the probability distribution of the output Defocus value, the maximum probability interval is found, and the predicted Defocus value is calculated. The closer the predicted Defocus value is to the target value (Groundtruth), the higher the confidence is, and the farther the predicted Defocus value is from the target value (Groundtruth), the lower the confidence is.

[0101] Figure 6a~6d It is a schematic diagram of the relationship between Defocus, confidence and DAC provided by an embodiment of the present invention. Figure 6a~6d It is a schematic diagram of the relationship between Defocus, confidence and DAC calculated by the PDAF confidence calculation method based on deep learning described in the present invention. Figure 6a~6d In the figure, the horizontal axis represents DAC, the vertical axis represents Defocus (please refer to the vertical axis on the left) and confidence (please refer to the vertical axis on the right), curve 1 represents the target curve (Groundtruth), curve 2 represents the Defocus curve, and curve 3 represents the confidence curve.

[0102] Please refer to Figure 6a As shown in the figure, the Defocus curve is very close to the target curve, so the confidence curve value is relatively high and stable. Figure 6bAs shown, only in the middle part of the horizontal axis is the Defocus curve close to the target curve. In this part, the confidence curve value is relatively high and relatively stable. In the left and right parts of the horizontal axis, there is a large deviation between the Defocus curve and the target curve. Therefore, in this part, the confidence curve value is relatively low and fluctuates greatly. Please refer to Figure 6c and 6d As shown in FIG. 1 , when the Defocus curve is close to the target curve, the confidence curve value is relatively high and relatively stable. When there is a relatively large deviation between the Defocus curve and the target curve, the confidence curve value is relatively low and the fluctuation is relatively large. Figure 6a~6d It can be concluded that the confidence calculated by the PDAF confidence calculation method based on deep learning described in the present invention is reliable.

[0103] The present invention trains the PDAF classification network parameters and can output not only the Defocus value but also the corresponding confidence level according to the probability distribution output by the PDAF classification network model, thereby effectively avoiding the focusing error caused by the CNN PDAF prediction error.

[0104] Figure 7 is a flowchart of calculating confidence using a PDAF confidence calculation method based on deep learning provided by an embodiment of the present invention. Figure 7 As shown, first input the image into the PDAF split network model, calculate the confidence, that is, use the PDAF confidence calculation method based on deep learning described in the present invention to calculate the confidence, and then judge the confidence. If the confidence value is relatively high, output the confidence (that is, the original output of PDAC, that is, output the original value of the calculated confidence). If the confidence value is relatively low, the PDAF output is set to zero or the confidence is further optimized. Exemplarily, a confidence target value or target range can be set. If the calculated confidence is higher than the target value or target range, the PDAF original output is executed. If the calculated confidence is lower than the target value or target range, the PDAF output is set to zero or further optimized.

[0105] In summary, in the PDAF confidence calculation method based on deep learning provided by the present invention, image data is first collected and data preprocessing is performed, and a PDAF classification network design is performed at the same time, and then the PDAF classification network parameters are trained to obtain a PDAF classification network model, and then the PD image is input to the PDAF classification network model, and the probability distribution of the Defocus value is output, and the Defocus value and the confidence are output according to the probability distribution of the Defocus value. The present invention can not only output the Defocus value but also the corresponding confidence by training the PDAF classification network parameters according to the probability distribution output by the PDAF classification network model, which can effectively avoid the focus error caused by the CNN PDAF prediction error.

[0106] The above description is only a description of the preferred embodiment of the present invention, and is not any limitation on the scope of rights of the present invention. Any technical personnel in this field can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A PDAF confidence calculation method based on deep learning, characterized in that: include: The image data is collected and preprocessed, and a PDAF classification network is designed at the same time; wherein the method for preprocessing the data includes generating a Defocus label and a confidence label; the method for designing the PDAF classification network includes: establishing a CNN network model; extracting feature information of the CNN network; and connecting a fully connected layer and a normalized exponential function; Training PDAF classification network parameters to obtain a PDAF classification network model; wherein the method for training PDAF classification network parameters includes: using a data enhancement method to further expand the data set to improve the effect of the model; using uniform distribution initialization; using a KL-divergence loss function to measure the approximate complex target distribution; Inputting the PD image into the PDAF classification network model, and outputting the probability distribution of Defocus values; and Output the Defocus value and the confidence level according to the probability distribution of the Defocus value; wherein the confidence level is calculated according to the probability distribution of the output Defocus value and the expected distribution, and the maximum probability interval is found according to the probability distribution of the output Defocus value to calculate the predicted Defocus value.

2. The PDAF confidence calculation method based on deep learning according to claim 1, characterized in that: The scenes for collecting image data include normal scenes, high-frequency scenes, moiré scenes and / or dark light scenes.

3. The PDAF confidence calculation method based on deep learning according to claim 1, characterized in that: The collected image data includes PD images, which are divided into 6*9 image blocks with pixels of 100*100*2, where *2 indicates that the training image PD data is split into left and right images.

4. The PDAF confidence calculation method based on deep learning according to claim 1, characterized in that: Methods for generating Defocus tags include: Calculate the contrast of the acquired image; Find Z points near the maximum gradient value and calculate the focus point position, where Z is a positive integer; and A Defocus tag is generated according to the focus point position.

5. The PDAF confidence calculation method based on deep learning according to claim 4, characterized in that: The calculation formula of the contrast of the image is: G x (x,y)=2I(x,y)-I(x-1,y)-I(x+1,y) G y (x,y)=2I(x,y)-I(x,y-1)-I(x,y+1) Here, x and y refer to the position of the pixel in the image.

6. The PDAF confidence calculation method based on deep learning according to claim 4, characterized in that: Defocus tags are: Label=DAC current -DAC focus Among them, DAC current Refers to the DAC value of the existing acquired image. DAC focus It refers to the DAC value corresponding to the focus point position, and DAC refers to the position where the motor moves.

7. The PDAF confidence calculation method based on deep learning according to claim 6, characterized in that: Methods for generating confidence labels include: Determine the Label interval range and Confidence_Label value, where Confidence_Label = Label + 499; Determine, according to the Confidence_Label value, the interval in which the Label value corresponding to each of the images falls and the probability of falling within the interval; and Generate confidence labels.

8. The PDAF confidence calculation method based on deep learning according to claim 1, characterized in that: According to the characteristics of the image data, the first layer is set to a two-channel input with a size of height*width*2.

9. The PDAF confidence calculation method based on deep learning according to claim 1, characterized in that: Extracting feature information of the CNN network includes: extracting L feature layers of different depths in the network, passing through 1*1 convolutional layers and fully connected layers respectively, obtaining M-dimensional feature vectors, and adding the offsets Dx and Dy from the center point to form L*M+2 dimensional features.

10. The PDAF confidence calculation method based on deep learning according to claim 1, characterized in that: The approximate complexity of the target distribution is measured using the following formula: Among them, p is the original probability distribution and q is the approximate probability distribution.

11. The PDAF confidence calculation method based on deep learning according to claim 1, characterized in that: The method for training the PDAF classification network parameters also includes: selecting the model with the best performance in the actual test based on the two models with the smallest loss function and the highest classification accuracy in the validation set.

Images