A deep learning-based PDAF model migration method

Abstract

The application provides a PDAF model migration method based on deep learning, comprising: training a network under a target module dataset to obtain a target model; calibrating physical differences between an existing module of the same type and the target module to obtain a correction module; and migrating the target model to the existing module and correcting according to the correction module. The application can migrate the target model trained based on the target module to the existing module of the same type, reduces the repeated data acquisition and model training process, and greatly improves the model reusability and algorithm reference efficiency.

Description

Technical Field

[0001] This invention relates to the field of camera imaging technology, and in particular to a deep learning-based PDAF model transfer method. Background Technology

[0002] Traditional focusing methods use contrast-detection autofocus, which uses the contrast of the image as a reference standard. During the focusing process, the point with the highest contrast is selected as the point of focus. The motor drives the lens to move from the lowest point to the highest point, recording the contrast of the image at each position. After the lens has moved the entire distance, it is moved back to the position with the highest contrast, thus completing the contrast-detection autofocus.

[0003] The basic principle of PDAF (Phase Detection Auto Focus) is that the part of the mobile phone module that senses the image is the photosensitive chip. Each pixel senses the image. If two symmetrical pixels that are a certain distance apart are covered respectively, covering the left and right halves of the pixel, it is equivalent to the left and right eyes of a person. Based on the different angles at which the two eyes see the object, the focus can be calculated to determine whether the focus is accurate.

[0004] Compared to contrast-detection autofocus, PDAF does not involve a full-process motor search. Instead, it determines the distance the motor needs to reach the point of focus by judging the phase difference, and then pushes the lens to that point of focus in one go, thus greatly improving focusing speed.

[0005] However, directly applying the CNN (Convolutional Neural Network) PDAF model trained on target module A to other similar sensor modules B results in a degradation of PDAF focusing performance. Reconstructing the dataset, generating labels, adjusting the network model, and training the network for different sensor modules would severely limit the algorithm's adaptability.

[0006] Patent CN111263070B provides a PDAF optimization method and device, but this patented method requires knowledge of the parameters of all lenses. The optimization methods differ for different parameters and different modules, and cannot guarantee universality. Charles Herrmann published a paper in 2022, "Learning to Autofocus," but this method cannot apply the model trained by module A to data different from that of module A. Summary of the Invention

[0007] The purpose of this invention is to provide a deep learning-based PDAF model transfer method that can transfer a target model trained on a target module to other modules of the same type, reducing the process of repeated data collection and model training, and greatly improving the reusability of the model and the efficiency of algorithm application.

[0008] To address the aforementioned technical problems, this invention provides a deep learning-based PDAF model transfer method, comprising:

[0009] Train the network on the target module dataset to obtain the target model;

[0010] The physical differences between existing modules of the same type and the target module are calibrated to obtain a correction module; and

[0011] The target model is migrated to the existing module and modified according to the modification module.

[0012] Optionally, methods for identifying physical differences between existing modules of the same type and the target module include:

[0013] Collect calibration images of existing modules and divide the calibration images of existing modules into multiple small windows;

[0014] The target value of the small window is calibrated, and the network prediction value of the small window is obtained to calculate the correction parameters of the small window;

[0015] Based on the correction parameters of the small windows contained in the test window, and the positional relationship between the test window and the small windows, bilinear interpolation is performed to calculate the correction parameters of the test window.

[0016] Optionally, the collected calibration images of the existing module include a checkerboard scene; the calibration images of the existing module are divided into 16*12 small windows with a resolution of 50*50 pixels each.

[0017] Optionally, the step of calibrating the target value of the small window and obtaining the network prediction value of the small window to calculate the correction parameters of the small window includes:

[0018] The target value of the small window is determined by contrast, and the linear parameters k2 and b2 of the target value are obtained.

[0019] Obtain the network prediction value of the small window, and fit the network prediction value into a straight line to obtain the straight line parameters k1 and b1;

[0020] Calculate the correction parameters k and b of the small window, where the formulas for calculating k and b are as follows:

[0021]

[0022] Optionally, the linear parameters of the target value are k2 = 1 and b2 = -target value.

[0023] Optionally, the correction module includes a calculation formula:

[0024] predit new =K*predit base +B;

[0025] Where predit is the predicted value output by the network. base This is the original predicted value, predit. new These are the corrected predicted values, and K and B are the correction parameters for the test window.

[0026] Optionally, the steps to train the network on the target module dataset to obtain the target model include:

[0027] Obtain the training set for the target module; perform data preprocessing; build the network model; train the network; obtain the target model.

[0028] Optionally, the method for migrating the target model to an existing module includes:

[0029] Obtain a test set for the existing module; apply the test set to the target model; obtain the prediction value before correction.

[0030] Optionally, the evaluation parameters of the PDAF model transfer method include curve correlation (PCC), zero-point accuracy (Offset), and overall error (Mean_error).

[0031] The formula for curve correlation PCC is:

[0032]

[0033] The formula for zero-point precision offset is:

[0034] Offset = |ZeroOffset PDAF -ZeroOffset CDAF |;

[0035] The formula for the overall error Mean_error is:

[0036]

[0037] Where DAC is the motor's moving position, DF is the DAC required to move to the focus, Cov is the covariance, D is the variance, and ZeroOffset is the zero offset. PDAF ZeroOffset is used to correct the zero-point accuracy of the focus point. CDAF The zero-point precision of the target value.

[0038] Optionally, the curve correlation PCC is used to evaluate the correlation between the DAC and DF based on 21 points near the focal point, and the zero-point accuracy Offset is used to evaluate the zero-point accuracy based on 21 points near the focal point.

[0039] In summary, the deep learning-based PDAF model transfer method provided by this invention involves training a network on a target module dataset to obtain a target model; identifying the physical differences between existing modules of the same type and the target module to obtain a correction module; transferring the target model to the existing module; and performing corrections based on the correction module. This invention can transfer a target model trained on a target module to an existing module of the same type, reducing repetitive data collection and model training processes, and greatly improving model reusability and algorithm application efficiency. Attached Figure Description

[0040] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention. Wherein:

[0041] Figure 1 This is a flowchart of a deep learning-based PDAF model transfer method provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of a calibration image provided in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram illustrating the relationship between Defocus and DAC provided in an embodiment of the present invention.

[0044] Figure 4 This is a flowchart of a deep learning-based PDAF model transfer method provided in an embodiment of the present invention.

[0045] Figure 5a This is a schematic diagram of the calibration image provided in Embodiment 1 of the present invention.

[0046] Figure 5b This is a schematic diagram illustrating the relationship between Defocus and DAC provided in Embodiment 1 of the present invention.

[0047] Figure 6a This is a schematic diagram of the calibration image provided in Embodiment 2 of the present invention.

[0048] Figure 6b This is a schematic diagram illustrating the relationship between Defocus and DAC provided in Embodiment 2 of the present invention.

[0049] Figure 7a This is a schematic diagram of the calibration image provided in Embodiment 3 of the present invention.

[0050] Figure 7b This is a schematic diagram illustrating the relationship between Defocus and DAC provided in Embodiment 3 of the present invention.

[0051] Figure 8a This is a schematic diagram of the calibration image provided in Embodiment 4 of the present invention.

[0052] Figure 8b This is a schematic diagram illustrating the relationship between Defocus and DAC provided in Embodiment 4 of the present invention. Detailed Implementation

[0053] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to facilitate and clarify the explanation of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different figures may emphasize different aspects and may sometimes use different scales.

[0054] As used in this invention, the singular forms “a,” “an,” and “the” include plural objects; the term “or” is generally used to mean “and / or”; the term “a number” is generally used to mean “at least one”; the term “at least two” is generally used to mean “two or more”; furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined as "first," "second," and "third" may explicitly or implicitly include one or at least two of those features. The term "proximal" usually refers to the end closer to the operator, and the term "distal" usually refers to the end closer to the patient. "One end" and "the other end," as well as "proximal" and "distal," usually refer to two corresponding parts, which include not only the endpoints. The terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can be fixed connections, detachable connections, or integral connections; they can be mechanical connections or electrical connections; they can be direct connections or indirect connections through an intermediate medium; they can be internal connections between two components or the interaction between two components.

[0055] Furthermore, as used in this invention, the phrase "one element is disposed on another element" generally only indicates that there is a connection, coupling, cooperation, or transmission relationship between the two elements, and the connection, coupling, cooperation, or transmission between the two elements can be direct or indirect through an intermediate element. It should not be construed as indicating or implying a spatial positional relationship between the two elements, i.e., one element can be located arbitrarily inside, outside, above, below, or to one side of the other element, unless otherwise explicitly stated. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0056] Figure 1 This is a flowchart of a deep learning-based PDAF model transfer method provided in an embodiment of the present invention. Please refer to... Figure 1 As shown, this invention provides a deep learning-based PDAF model transfer method, comprising:

[0057] S1: Train the network on the target module dataset to obtain the target model;

[0058] S2: Obtain the correction module by calibrating the physical differences between existing modules of the same type and the target module;

[0059] S3: Migrate the target model to the existing module and make corrections according to the correction module.

[0060] Specifically, in step S1, the network is trained on the target module dataset to obtain the target model. For example, first, the training set of the target module (e.g., module A) is obtained, then the data is preprocessed, such as by normalization, then a network model is built based on the preprocessed data, then the network model is trained, such as by adjusting parameters and optimizing model performance, and finally the target model (e.g., target model A) is obtained.

[0061] In step S2, the physical differences between existing modules of the same type and the target module are calibrated to obtain the correction module.

[0062] It should be noted that the existing module and the target module are of the same type. If they are not of the same type, the PDAF model migration method described in this invention cannot be used.

[0063] For example, methods for identifying physical differences between existing modules of the same type and target modules include:

[0064] First, calibration images of existing modules are acquired and divided into multiple small windows. In this embodiment, the acquired calibration images of existing modules generally use a checkerboard pattern because the network prediction curve of a checkerboard pattern has good linearity and high accuracy, but this is not a limitation. The calibration images of existing modules are divided into multiple small windows; preferably, they are divided into 16*12 pixels, each 50*50 pixels wide. Please refer to [reference needed]. Figure 2 As shown, of course, it can be divided into small windows of other numbers of pixels, and the present invention does not limit this.

[0065] Next, the target value of the small window is calibrated, and the network prediction value of the small window is obtained to calculate the correction parameters of the small window. Figure 3 This is a schematic diagram illustrating the relationship between Defocus and DAC according to an embodiment of the present invention, where the horizontal axis represents DAC (motor movement position) and the vertical axis represents Defocus (DAC required to move to focus). Please refer to... Figure 3 As shown, firstly, the target value (Label value) of the small window is determined by contrast, and the linear parameters k2 and b2 of the target value are obtained, where k2 = 1 and b2 = -target value, i.e., b2 = -target value. Figure 3 The dashed line in the diagram represents the result under the target module. Next, the network prediction values ​​of the small window are obtained, and these values ​​are fitted to a straight line to obtain the line parameters k1 and b1. Specifically, the network prediction values ​​of the small window DAC[value 1, value 2] are obtained, and the network prediction values ​​of DAC[value 1, value 2] are fitted to a straight line, as shown below. Figure 3 The solid line in the diagram represents the network prediction result. The line parameters k1 and b1 can be obtained from Defocus = k1 * DAC + b1. Finally, the correction parameters k and b for the small window are calculated, where the formulas for k and b are as follows:

[0066]

[0067] Then, based on the correction parameters of the small windows contained in the test window and the positional relationship between the test window and the small windows, bilinear interpolation is performed to calculate the correction parameters K and B of the test window.

[0068] The final correction module includes the calculation formula: predit new =K*predit base +B, where predit is the predicted value output by the network. base This is the original predicted value, predit. new These are the corrected predicted values, and K and B are the correction parameters for the test window.

[0069] In step S3, the target model is migrated to an existing module and corrected according to the correction module. Specifically, first, a test set of the existing module (e.g., module B) is obtained, and then the test set is applied to the target model (e.g., target model A); then, the predicted value before correction, i.e., predit, is obtained. base .

[0070] Then, the prediction value predit is corrected according to the correction module. base Apply to formula predit new =K*predit base In +B, based on the already calculated correction parameters K and B of the test window, the corrected prediction value predit is calculated. base .

[0071] Figure 4 This is a flowchart of a deep learning-based PDAF model transfer method provided in an embodiment of the present invention. Please refer to it. Figure 4 As shown, firstly, the network is trained on the target module dataset to obtain the target model. That is, the training set of module A is obtained first, followed by data preprocessing, then building the network model, and finally training the network, such as adjusting parameters and optimizing model performance, to obtain the target model A. Next, the physical differences between existing modules of the same type and the target module are calibrated to obtain the correction module. That is, the calibration parameters of module B are obtained, data calibration is performed, then small windows k and b are calculated, and finally test windows K and B are calculated to obtain the correction module. Next, the test set of module B is obtained, and the test set is applied to the target model A to obtain the network prediction value (before correction). Applying the network prediction value to the correction module yields the corrected result.

[0072] The deep learning-based PDAF model transfer method provided by this invention involves training a network on a target module dataset to obtain a target model; identifying the physical differences between existing modules of the same type and the target module to obtain a correction module; transferring the target model to the existing module and correcting it according to the correction module. This invention can transfer a target model trained on a target module to an existing module of the same type, reducing the process of repeated data collection and model training, and greatly improving model reusability and algorithm application efficiency.

[0073] The evaluation parameters of the PDAF model transfer method include curve correlation (PCC), zero-point accuracy (Offset), and overall error (Mean_error).

[0074] The formula for curve correlation PCC is:

[0075]

[0076] The formula for zero-point precision offset is:

[0077] Offset = |ZeroOffset PDAF -ZeroOffset CDAF |;

[0078] The formula for the overall error Mean_error is:

[0079]

[0080] Where DAC is the motor's moving position, DF is the DAC required to move to the focus, Cov is the covariance, D is the variance, and ZeroOffset is the zero offset. PDAF ZeroOffset is used to correct the zero-point accuracy of the focus point. CDAF The zero-point precision of the target value.

[0081] It should be noted that the curve correlation PCC is based on 21 points near the focal point to evaluate the correlation between DAC and DF, and the zero-point accuracy Offset is based on 21 points near the focal point to evaluate the zero-point accuracy.

[0082] The following specific embodiments verify the deep learning-based PDAF model transfer method provided by the present invention.

[0083] Figure 5a This is a schematic diagram of the calibration image provided in Embodiment 1 of the present invention. Figure 5b This is a schematic diagram illustrating the relationship between Defocus and DAC provided in Embodiment 1 of the present invention. According to... Figure 5a The calibration images shown are used to obtain the final evaluation parameters, which are shown in Table 1.

[0084] Table 1

[0085] PCC Offset Mean_error Before revision 0.998 6.423 8.807 Revised 0.998 2.895 7.389

[0086] As can be seen from Table 1, compared with before the correction, both the zero-point precision offset and the overall error Mean_error have decreased after the correction, especially the zero-point precision offset, which has decreased significantly. Figure 5b In the diagram, curve 1 represents the target curve, curve 2 represents the curve before correction, and curve 3 represents the curve after correction. From... Figure 5b As can be seen, compared to curve 2, curve 3 is closer to curve 1, that is, the corrected curve obtained by the deep learning-based PDAF model transfer method described in this invention is closer to the target curve.

[0087] Figure 6aThis is a schematic diagram of the calibration image provided in Embodiment 2 of the present invention. Figure 6b This is a schematic diagram illustrating the relationship between Defocus and DAC provided in Embodiment 2 of the present invention. Figure 6a The calibration images shown are used to obtain the final evaluation parameters, which are shown in Table 2.

[0088] Table 2

[0089] PCC Offset Mean_error Before revision 0.998 5.018 6.090 Revised 0.998 1.386 4.390

[0090] As can be seen from Table 2, compared with before the correction, both the zero-point precision Offset and the overall error Mean_error have decreased after the correction, especially the zero-point precision Offset has decreased significantly. Figure 6b In the diagram, curve 1 represents the target curve, curve 2 represents the curve before correction, and curve 3 represents the curve after correction. From... Figure 6b As can be seen, compared to curve 2, curve 3 is closer to curve 1, that is, the corrected curve obtained by the deep learning-based PDAF model transfer method described in this invention is closer to the target curve.

[0091] Figure 7a This is a schematic diagram of the calibration image provided in Embodiment 3 of the present invention. Figure 7b This is a schematic diagram illustrating the relationship between Defocus and DAC provided in Embodiment 3 of the present invention. Figure 7a The calibration images shown are used to obtain the final evaluation parameters, which are shown in Table 3.

[0092] Table 3

[0093] PCC Offset Mean_error Before revision 0.999 2.055 8.088 Revised 0.999 1.895 3.873

[0094] As can be seen from Table 3, compared with before the correction, both the zero-point accuracy Offset and the overall error Mean_error have decreased after the correction, especially the overall error Mean_error has decreased significantly. Figure 7b In the diagram, curve 1 represents the target curve, curve 2 represents the curve before correction, and curve 3 represents the curve after correction. From... Figure 7b As can be seen, compared to curve 2, curve 3 is closer to curve 1, that is, the corrected curve obtained by the deep learning-based PDAF model transfer method described in this invention is closer to the target curve.

[0095] Figure 8a This is a schematic diagram of the calibration image provided in Embodiment 4 of the present invention. Figure 8b This is a schematic diagram illustrating the relationship between Defocus and DAC provided in Embodiment 4 of the present invention. Figure 8a The calibration images shown are used to obtain the final evaluation parameters, which are shown in Table 4.

[0096] Table 4

[0097] PCC Offset Mean_error Before revision 0.998 4.856 6.824 Revised 0.998 1.383 4.755

[0098] As can be seen from Table 4, compared with the original, both the zero-point accuracy Offset and the overall error Mean_error have decreased after the correction, and the decrease is relatively large. Figure 8b In the diagram, curve 1 represents the target curve, curve 2 represents the curve before correction, and curve 3 represents the curve after correction. From... Figure 8b As can be seen, compared to curve 2, curve 3 is closer to curve 1, and curve 3 and curve 1 basically overlap in the latter half. That is, the corrected curve obtained by using the deep learning-based PDAF model transfer method described in this invention is closer to the target curve.

[0099] As can be seen from the evaluation parameter calculation results of the above embodiments and the relationship diagram between Defocus and DAC, the PDAF effect after transfer using the deep learning-based PDAF model transfer method described in this invention meets expectations.

[0100] In summary, the deep learning-based PDAF model transfer method provided by this invention involves training a network on a target module dataset to obtain a target model; identifying the physical differences between existing modules of the same type and the target module to obtain a correction module; transferring the target model to the existing module; and performing corrections based on the correction module. This invention can transfer a target model trained on a target module to an existing module of the same type, reducing repetitive data collection and model training processes, and greatly improving model reusability and algorithm application efficiency.

[0101] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A deep learning-based PDAF model transfer method, characterized in that, include: Train the network on the target module dataset to obtain the target model; The physical differences between existing modules of the same type and the target module are calibrated to obtain the correction module; as well as The target model is migrated to the existing module and corrected according to the correction module; The methods for identifying the physical differences between existing modules of the same type and the target module include: Collect calibration images of existing modules and divide the calibration images of existing modules into multiple small windows; The target value of the small window is calibrated, and the network prediction value of the small window is obtained to calculate the correction parameters of the small window; Based on the correction parameters of the small windows contained in the test window, and the positional relationship between the test window and the small windows, bilinear interpolation is performed to calculate the correction parameters of the test window.

2. The deep learning-based PDAF model transfer method according to claim 1, characterized in that, The collected calibration images of the existing module contain a checkerboard scene; the calibration images of the existing module are divided into 16*12 small windows with a resolution of 50*50 pixels each.

3. The deep learning-based PDAF model transfer method according to claim 1, characterized in that, The steps of calibrating the target value of the small window and obtaining the network prediction value of the small window to calculate the correction parameters of the small window include: The target value of the small window is determined by contrast, and the linear parameters k2 and b2 of the target value are obtained. Obtain the network prediction value of the small window, and fit the network prediction value into a straight line to obtain the straight line parameters k1 and b1; Calculate the correction parameters k and b of the small window, where the formulas for calculating k and b are as follows: 。 4. The deep learning-based PDAF model transfer method according to claim 3, characterized in that, The linear parameters of the target value are k2 = 1 and b2 = -target value.

5. The deep learning-based PDAF model transfer method according to claim 3, characterized in that, The correction module includes a calculation formula: ； in, It is the predicted value output by the network. This is the predicted value before correction. These are the corrected predicted values, and K and B are the correction parameters for the test window.

6. The deep learning-based PDAF model transfer method according to claim 1, characterized in that, The steps to train the network on the target module dataset and obtain the target model include: Obtain the training set for the target module; perform data preprocessing; build the network model; train the network; obtain the target model.

7. The deep learning-based PDAF model transfer method according to claim 1, characterized in that, Methods for migrating the target model to an existing module include: Obtain a test set for the existing module; apply the test set to the target model; obtain the prediction value before correction.

8. The deep learning-based PDAF model transfer method according to claim 1, characterized in that, The evaluation parameters of the PDAF model transfer method include curve correlation (PCC), zero-point accuracy (Offset), and overall error (Mean_error). The formula for curve correlation PCC is: ； The formula for zero-point precision offset is: ； The formula for the overall error Mean_error is: ； Where DAC is the motor's moving position, DF is the DAC required to move to the focus, Cov is the covariance, and D is the variance. To correct the zero-point accuracy of the focus point, The zero-point precision of the target value.

9. The deep learning-based PDAF model transfer method according to claim 8, characterized in that, Curve correlation (PCC) is used to evaluate the correlation between DAC and DF based on 21 points near the focal point, while zero-point accuracy (Offset) is used to evaluate zero-point accuracy based on 21 points near the focal point.

Images