Method, device and storage medium for generating training data of auto-focus model

Abstract

The present application provides a method, device and storage medium for generating training data of an autofocus model, the method comprising: obtaining a first slope obtained by linearly fitting multiple simulated two-dimensional data consisting of a simulated PSF radius of a first defocused image and a first defocus amount, and a second slope obtained by linearly fitting multiple actual two-dimensional data consisting of a sample PSF radius of an actual sample image and a sample defocus amount; simulating wavefront distortion according to the tolerance range of an image acquisition device; obtaining multiple second defocused images and a second defocus amount based on Fraunhofer diffraction simulation; correcting the second defocus amount based on the first slope and the second slope to obtain a third defocus amount, and determining the second defocus image and the third defocus amount as training data. The method can achieve efficient and accurate autofocus under a small number of sample conditions by constructing a defocus optical model and using the model to generate a large amount of training data with defocus information.

Description

Technical Field

[0001] The present application relates to the field of autofocus, and in particular, to a method for generating training data of an autofocus model, a device for generating training data of an autofocus model, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the development of computer vision and digital imaging technology, autofocus technology has become an important part of image processing and machine vision. Traditional autofocus methods usually rely on a large amount of sample data to train the model, which is not only time-consuming but also costly. In addition, in some specific application scenarios, it is very difficult to obtain a large number of high-quality training samples, which will result in low accuracy of the trained model. Summary of the invention

[0003] The main purpose of the present application is to provide a method for generating training data of an autofocus model, a device for generating training data of an autofocus model, a computer-readable storage medium and an electronic device, so as to at least solve the problem that the autofocus method in the prior art relies on a large amount of sample data to train the model, which is not only time-consuming but also costly.

[0004] To achieve the above-mentioned purpose, according to one aspect of the present application, a method for generating training data of an autofocus model is provided, comprising: obtaining a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, one of the simulated two-dimensional data comprising a simulated PSF radius and a first defocus amount of a first defocus image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, one of the actual two-dimensional data comprising a sample PSF radius and a sample defocus amount of an actual sample image, the actual sample image is a sample image actually collected, and the first defocus image is a defocus image obtained by simulation; determining various assembly tolerances and tolerance ranges of an image acquisition device, and simulating the wavefront distortion of the image acquisition device according to the tolerance range; obtaining a plurality of second defocus images based on the wavefront distortion and Fraunhofer diffraction simulation, and determining a second defocus amount corresponding to the second defocus image; correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount are used for training the autofocus model.

[0005] Optionally, obtaining the first slope includes: based on the image imaging principle and optical diffraction simulation, obtaining a plurality of first defocus images of the image acquisition device according to different first defocus amount simulations, wherein the first defocus images correspond one-to-one to the first defocus amounts; respectively calculating the PSF radius of each of the first defocus images to obtain the simulated PSF radius corresponding to each of the first defocus images, and determining a group of corresponding first defocus amounts and simulated PSF radii as one of the simulated two-dimensional data, wherein the first defocus images correspond one-to-one to the simulated two-dimensional data; calculating the first mean value according to the formula , calculate the first radius mean, where is the simulated PSF radius of the first defocused image, is the simulated PSF radius of the second first defocused image, is the simulated PSF radius of the mth first defocused image, is the first radius mean, m is the number of the first defocused images; according to all the first defocus amounts and the second mean calculation formula , calculate and obtain the first defocus amount mean; is the first defocus amount of the first first defocus image, is the first defocus amount of the second first defocus image, is the first defocus amount of the mth first defocus image, is the first defocus amount mean; according to the first radius mean, the first defocus amount mean, all the simulation two-dimensional data and the first slope calculation formula , calculate and obtain the first slope, where G1 is the first slope.

[0006] Optionally, obtaining the second slope includes: collecting a plurality of actual sample images and obtaining a sample defocus amount of each of the actual sample images; calculating the PSF radius of each of the actual sample images respectively to obtain a sample PSF radius corresponding to each of the actual sample images, and determining a set of corresponding sample defocus amounts and sample PSF radii as one of the actual two-dimensional data, wherein the actual sample images correspond to the actual two-dimensional data one by one; calculating the sample PSF radius according to all of the sample PSF radii and the third mean value formula , calculate the second radius mean, where is the sample PSF radius of the first actual sample image, is the sample PSF radius of the second actual sample image, is the sample PSF radius of the nth actual sample image, is the second radius mean, n is the number of the actual sample images; according to all the sample defocus amounts and the fourth mean calculation formula , calculate and obtain the second defocus amount mean; is the sample defocus amount of the first actual sample image, is the sample defocus amount of the second actual sample image, is the sample defocus amount of the nth actual sample image, is the second defocus amount mean; according to the second radius mean, the second defocus amount mean, all the actual two-dimensional data and the second slope calculation formula , the second slope is calculated, where G2 is the second slope.

[0007] Optionally, simulating the wavefront distortion of the image acquisition device according to the tolerance range includes: simulating the change of the image acquisition device within the tolerance range according to the various types of assembly tolerances and the tolerance range, calculating the wavefront difference once for each change using optical design software and recording the first NZ Zernike coefficients corresponding to the current wavefront difference, where NZ is an integer greater than or equal to a preset value; calculating the first-order correlation coefficient between any type of the assembly tolerance and any one of the Zernike coefficients, and determining the Zernike coefficient with the highest correlation with each of the assembly tolerances as the target Zernike coefficient based on all the first-order correlation coefficients, and determining the fluctuation range of the target Zernike coefficient; randomly generating multiple groups of coefficients based on all the target Zernike coefficients and the corresponding fluctuation ranges, wherein a group of the coefficients includes NZ vectors, and obtaining coefficients of a Zernike polynomial, wherein the Zernike polynomial has NZ terms, wherein the target Zernike coefficient in a group of the coefficients is within the corresponding fluctuation range; and calculating the Zernike polynomial according to the Zernike polynomial, the coefficients of the Zernike polynomial and the distortion formula. , determining the wavefront distortion of the image acquisition device, is the wavefront distortion, is the Zernike polynomial, are the coefficients of the Zernike polynomial.

[0008] Optionally, correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount includes: based on the first slope, the second slope and a correction formula , correct each of the second defocus amounts to obtain the corresponding third defocus amount, wherein G1 is the first slope, G2 is the second slope, is the third defocus amount, is the second defocus amount.

[0009] Optionally, after obtaining the corresponding third defocus amount, the method also includes: constructing a neural network model, the neural network model including at least a convolutional layer and a fully connected layer; training the neural network model using a training set based on a loss function, and stopping the training when the average loss value of the verification set is less than a preset value, so as to obtain a trained autofocus model, wherein both the training set and the verification set include data consisting of the second defocus image and the corresponding third defocus amount, and the data in the training set is different from the data in the verification set.

[0010] Optionally, after obtaining the trained autofocus model, the method further includes: acquiring a pre-prediction image and a sobel value of the pre-prediction image, and inputting the pre-prediction image into the autofocus model to obtain a predicted defocus amount; adjusting the position of a photosensitive chip in an image acquisition device according to the predicted defocus amount, and using the image acquisition device to obtain a post-prediction image, and determining the sobel value of the post-prediction image; when the sobel value of the pre-prediction image is less than the sobel value of the post-prediction image, determining that the predicted defocus amount is accurate, and when the sobel value of the pre-prediction image is greater than the sobel value of the post-prediction image, determining that the predicted defocus amount is inaccurate, and deleting the predicted defocus amount.

[0011] According to another aspect of the present application, a device for generating training data of an autofocus model is provided, comprising: an acquisition unit, for acquiring a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one of the simulated two-dimensional data comprises a simulated PSF radius and a first defocus amount of a first defocused image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one of the actual two-dimensional data comprises a sample PSF radius and a sample defocus amount of an actual sample image, wherein the actual sample image is a sample image actually collected, and the first defocused image is a defocused image obtained by simulation; a determination unit, for determining various assembly tolerances and tolerance ranges of an image acquisition device, and simulating the wavefront distortion of the image acquisition device according to the tolerance range; a simulation unit, for obtaining a plurality of second defocused images based on the wavefront distortion and Fraunhofer diffraction simulation, and determining a second defocus amount corresponding to the second defocused image; a correction unit, for correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount, wherein the data consisting of the second defocus image and the corresponding third defocus amount is used for training the autofocus model.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for generating training data for the autofocus model.

[0013] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for generating training data for executing any one of the autofocus models.

[0014] Applying the technical solution of the present application, the method for generating training data of the above-mentioned autofocus model first obtains a first slope and a second slope, the first slope being obtained by linear fitting of multiple simulated two-dimensional data, one simulated two-dimensional data including a simulated PSF radius and a first defocus amount of a first defocus image; the second slope being obtained by linear fitting of multiple actual two-dimensional data, one actual two-dimensional data including a sample PSF radius and a sample defocus amount of an actual sample image; determining various assembly tolerances and tolerance ranges of the image acquisition device, and simulating the wavefront distortion of the image acquisition device according to the tolerance range; obtaining multiple second defocus images based on wavefront distortion and Fraunhofer diffraction simulation, and determining the second defocus amount corresponding to the second defocus image; correcting each second defocus amount based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount are used for training the autofocus model. This method constructs a defocus optical model and uses the model to generate a large amount of training data with defocus information. It can achieve efficient and accurate autofocus under a small number of sample conditions, solving the problem that the autofocus method in the existing technology relies on a large amount of sample data to train the model, which is not only time-consuming but also costly. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for generating training data of an auto-focus model provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram showing the principle of a through-focus imaging system is shown;

[0018] Figure 3A schematic diagram of a process for generating training data for an auto-focus model provided in accordance with an embodiment of the present application is shown;

[0019] Figure 4 A schematic diagram of a small amount of real defocused sample images collected according to Embodiment 1 of the present application is shown;

[0020] Figure 5 A schematic diagram of a simulated defocused image provided according to Example 1 of the present application is shown;

[0021] Figure 6 A schematic diagram of a loss function curve provided according to Example 1 of the present application is shown;

[0022] Figure 7 A schematic diagram of a process for generating training data for an auto-focus model provided in accordance with an embodiment of the present application is shown;

[0023] Figure 8 A structural block diagram of a device for generating training data for an auto-focus model provided in accordance with an embodiment of the present application is shown.

[0024] The above drawings include the following reference numerals:

[0025] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0030] The PSF (Point Spread Function) radius of an image is an important parameter to measure the size of a light spot when an optical system diffuses a point light source during imaging. PSF describes how an optical system maps an ideal point light source to the light intensity distribution on the imaging plane, and is a key indicator of optical imaging quality and system resolution.

[0031] Zernike coefficients are a set of mathematical parameters used in the field of optics to describe wavefront distortion. They are based on the theory of Zernike polynomials. Zernike polynomials are a set of orthogonal polynomials defined in a circular domain and are widely used to describe optical wavefronts, especially in imaging systems, laser systems, and astronomical telescopes, to analyze and correct wavefront aberrations.

[0032] As introduced in the background technology, the autofocus method in the prior art usually relies on a large amount of sample data to train the model, which is not only time-consuming but also costly. In addition, in some specific application scenarios, it is very difficult to obtain a large number of high-quality training samples. In order to solve the problem that the autofocus method in the prior art relies on a large amount of sample data to train the model, which is not only time-consuming but also costly, and the autofocus demand of the scene with few samples is difficult to be met, the embodiments of the present application provide a method for generating training data of an autofocus model, a device for generating training data of an autofocus model, a computer-readable storage medium and an electronic device.

[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0034] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 11 is a hardware structure block diagram of a mobile terminal of a method for generating training data of an auto-focus model according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0035] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for generating training data of the auto-focus model in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] In this embodiment, a method for generating training data for an autofocus model running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0037] For Figure 2 For the general imaging system shown in the figure, the ideal image point corresponding to the object point P after passing through the imaging system is , when the photosensitive chip is located When the object point P is captured, the image point P is the clearest. The radius is the smallest. When the photosensitive chip deviates from the ideal image point When the actual image point The theoretical calculation formula of the radius R is: , where D is the pupil diameter of the imaging system, is the object distance, is the effective focal length of the imaging system, Indicates the degree of deviation of the actual image plane from the ideal image plane, that is, the defocus amount, v is the image position after deviation, and v0 is the ideal image position. In the mass production and assembly process of the lens, D, , It can be considered as a constant optical parameter with small perturbations. The theoretical calculation formula of radius R shows that the relationship between R and d is linear, defining the characteristic slope .

[0038] Figure 3 FIG. 1 is a flow chart of a method for generating training data for an auto-focus model according to an embodiment of the present application. Figure 3 As shown, the method comprises the following steps:

[0039] Step S201, obtaining a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one of the simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocused image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one of the actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image, wherein the actual sample image is a sample image actually collected, and the first defocused image is a defocused image obtained by simulation;

[0040] Specifically, on the one hand, through optical diffraction simulation, the ideal imaging image of the module can be obtained. By changing the defocus amount in the optical diffraction simulation and calculating the PSF of the simulated defocus image, a set of theoretical defocus feature data can be obtained. Subsequent processing of these theoretical defocus feature data can generate a large amount of effective training data in a short time. The cost of obtaining model training data is low, which is conducive to reducing the difficulty of model training in small sample scenarios. On the other hand, collecting a small number of real module defocus samples and subsequently obtaining a large amount of training data based on the real defocus samples can increase the accuracy of the model.

[0041] Wherein, obtaining the first slope includes the following steps:

[0042] Step S301, based on the image imaging principle and optical diffraction simulation, a plurality of first defocused images of the image acquisition device are obtained according to simulations of different first defocus amounts, wherein the first defocused images correspond to the first defocus amounts one by one;

[0043] Step S302, respectively calculating the PSF radius of each of the above-mentioned first defocused images, obtaining the simulated PSF radius corresponding to each of the above-mentioned first defocused images, and determining a set of corresponding first defocus amounts and the above-mentioned simulated PSF radius as the above-mentioned simulated two-dimensional data, and the above-mentioned first defocused images correspond to the above-mentioned simulated two-dimensional data one by one;

[0044] Step S303, based on all the above simulation PSF radii and the first mean value calculation formula , calculate the first radius mean, where is the simulated PSF radius of the first defocused image, is the simulated PSF radius of the second first defocused image, is the simulated PSF radius of the mth first defocused image, is the mean value of the first radius, and m is the number of the first defocused images;

[0045] Step S304, calculating the first defocus amount and the second mean value according to all the above formulas , calculate and obtain the first defocus amount mean; is the first defocus amount of the first defocus image, is the first defocus amount of the second first defocus image, is the first defocus amount of the mth first defocus image, is the average value of the first defocus amount;

[0046] Step S305, according to the first radius mean, the first defocus amount mean, all the above two-dimensional simulation data and the first slope calculation formula , the first slope is calculated, where G1 is the first slope.

[0047] Specifically, using Figure 2 Middle D, , By changing the defocus value in the optical diffraction simulation and calculating the PSF of the simulated defocus image, a set of theoretical defocus characteristic data (rt1, dt1), (rt2, dt2), ..., (rt m , dt m). To ensure accuracy, the value of m should not be less than 10. This set of theoretical defocus feature data is the above-mentioned multiple simulated two-dimensional data. The first slope can be obtained by processing this set of theoretical defocus feature data according to the above steps. Subsequently, based on the first slope, a large amount of effective training data can be generated in a short time.

[0048] Wherein, obtaining the second slope comprises the following steps:

[0049] Step S401, collecting a plurality of actual sample images and obtaining a sample defocus value of each of the actual sample images;

[0050] Step S402, respectively calculating the PSF radius of each of the actual sample images, obtaining the sample PSF radius corresponding to each of the actual sample images, and determining a set of corresponding sample defocus amounts and sample PSF radiuses as one of the actual two-dimensional data, wherein the actual sample images correspond to the actual two-dimensional data one by one;

[0051] Step S403, based on all the above sample PSF radii and the third mean calculation formula , calculate the second radius mean, where is the sample PSF radius of the first actual sample image, is the sample PSF radius of the second actual sample image, is the sample PSF radius of the nth actual sample image, is the mean value of the second radius, and n is the number of the actual sample images;

[0052] Step S403, calculating the defocus values ​​of all the above samples and the fourth mean value using the formula , calculate and obtain the second defocus amount mean; is the sample defocus amount of the first actual sample image, is the sample defocus amount of the second actual sample image, is the sample defocus amount of the nth actual sample image, is the second defocus amount average value;

[0053] Step S404, calculating the slope according to the second radius mean, the second defocus amount mean, all the actual two-dimensional data and the second slope calculation formula , the second slope is calculated, where G2 is the second slope.

[0054] Specifically, in the actual module production process, due to assembly and processing, D, u, and f will deviate from the design values. Therefore, the training data generated by simulating the imaging process based on the design values ​​cannot represent the real module and cannot be used for subsequent neural network training. Then it is necessary to collect a small number of real module defocus samples, record the defocus amount d, calculate the PSF radius R of the sample, and obtain a set of actual defocus feature data (r1, d1), (r2, d2), ..., (r n , d n ). To ensure accuracy, the value of n should be no less than 10. This set of actual defocus feature data is the above-mentioned multiple actual two-dimensional data. The second slope can be obtained by processing this set of actual defocus feature data according to the above steps. Subsequently, training data is generated based on the second slope, which can improve the accuracy of the training data, thereby greatly improving the accuracy of the model.

[0055] Step S202, determining various assembly tolerances and tolerance ranges of the image acquisition device, and simulating the wavefront distortion of the image acquisition device according to the tolerance ranges;

[0056] Specifically, wavefront distortion can be described by a linear combination of Zernike polynomials, each of which corresponds to a specific type of aberration. By calculating wavefront distortion, the Zernike coefficients can be obtained, and then the main types and degrees of aberrations in the optical system can be analyzed, providing data support for system design and correction.

[0057] Wherein, simulating and obtaining the wavefront distortion of the image acquisition device according to the tolerance range includes the following steps:

[0058] Step S2021, simulating the change of the image acquisition device within the tolerance range according to the various assembly tolerances and tolerance ranges, calculating the wavefront difference once for each change using optical design software and recording the first NZ Zernike coefficients corresponding to the current wavefront difference, where NZ is an integer greater than or equal to a preset value;

[0059] Step S2022, calculating the first-order correlation coefficient between any type of the above-mentioned assembly tolerance and any of the above-mentioned Zernike coefficients, and according to all the above-mentioned first-order correlation coefficients, determining the above-mentioned Zernike coefficient with the highest correlation with each of the above-mentioned assembly tolerances as the target Zernike coefficient, and determining the fluctuation range of the above-mentioned target Zernike coefficient;

[0060] The type of assembly tolerance is determined according to the lens structure, and may include but is not limited to the eccentricity, tilt, air gap of the lens, the distance between each lens group and the image sensor (also called a photosensitive chip), etc.

[0061] The correlation coefficient between each type of assembly tolerance and each Zernike term is calculated, and the Zernike terms with a high correlation with the assembly tolerance are selected so that the wavefront obtained in the subsequent simulation is closer to the real sample, thereby improving the validity of the training data and the reliability of the model.

[0062] Step S2023, randomly generating multiple groups of coefficients according to all the above target Zernike coefficients and the corresponding fluctuation ranges, wherein one group of the above coefficients includes NZ vectors, and obtaining coefficients of a Zernike polynomial, wherein the above Zernike polynomial has NZ terms, wherein the target Zernike coefficients in one group of the above coefficients are within the corresponding fluctuation range;

[0063] Step S2024, according to the Zernike polynomial, the coefficients of the Zernike polynomial and the distortion formula , determine the wavefront distortion of the above image acquisition device, is the above wavefront distortion, is the above Zernike polynomial, are the coefficients of the above Zernike polynomials.

[0064] Specifically, the currently commonly used defocused imaging simulation uses ray tracing and tolerance simulation, which is inefficient and cannot quickly generate a large amount of training data. Therefore, tolerance simulation can be used to obtain the Zernike coefficients and distortion formulas with high correlation to obtain wavefront distortion, so that a large amount of effective training data can be obtained based on this in the future.

[0065] The specific algorithm flow is as follows:

[0066] Step S1: Reconstruct the entire lens in the optical design software according to the optical lens design parameters, introduce the tolerance range of the lens assembly, so that the lens posture changes within the tolerance range, and calculate the wavefront difference once each time it changes using the optical design software and record the first NZ term Zernike coefficient corresponding to the wavefront difference. (NZ is not less than 15).

[0067] Step S2: Calculate the first-order correlation coefficient between any tolerance and any Zernike coefficient, find the Zernike coefficient with a high correlation with the tolerance, and calculate the fluctuation range of the corresponding Zernike coefficient.

[0068] Step S3: Based on the relevant Zernike terms and their corresponding Zernike coefficient fluctuation ranges obtained in step S2, a set of vectors containing NZ numbers is randomly generated, so that the coefficients of the Zernike terms with higher correlation vary randomly within the corresponding Zernike coefficient fluctuation range, and the coefficients of the other irrelevant Zernike terms are equal to a random minimum. In this way, the first NZ terms of the Zernike coefficients can be effectively simulated to simulate a wavefront with aberrations. , It is the wavefront distortion, which can usually be represented by a linear combination of a series of orthogonal polynomials, such as Zernike polynomials. The wavefront distortion at the pupil can be described by Zernike polynomials through the following formula: , are the coefficients of the polynomial. Accurately obtaining the coefficients of the Zernike polynomial can describe the optical aberration of the imaging system.

[0069] Step S203, obtaining a plurality of second defocused images based on the wavefront distortion and Fraunhofer diffraction simulation, and determining a second defocus amount corresponding to the second defocused images;

[0070] Specifically, by using detailed information on wavefront distortion and Fraunhofer diffraction theory, images under different defocus states can be accurately simulated. This simulation technology can generate a large number of realistic defocus images, overcoming the problems of low efficiency and slow data generation speed of traditional ray tracing and tolerance simulation.

[0071] Step S204, correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount is used for training an autofocus model.

[0072] Specifically, correcting the defocus amount generated by the simulation based on the first slope and the second slope can significantly improve the training data quality of the autofocus model, thereby improving the accuracy, generalization ability and training efficiency of the model, reducing dependence on actual samples, and thus improving the model availability in a small number of sample scenarios.

[0073] Among them, based on the above first slope, the above second slope and the correction formula , each of the above second defocus amounts is corrected to obtain the corresponding third defocus amount, wherein G1 is the above first slope, G2 is the above second slope, is the third defocus amount mentioned above, is the second defocus amount mentioned above.

[0074] In some embodiments, wavefront distortion is obtained based on the highly correlated Zernike coefficients, combined with the first slope G1 and the second slope G2, and Fraunhofer diffraction simulation is used to generate a large amount of effective training data in a short time.

[0075] That is, after the above step S3, the method further includes step S4: using Fraunhofer diffraction to simulate the light field at different defocus distances. The image is formed on the image plane. Using G1 and G2 calculated in the above steps, the defocus amount corresponding to the image is corrected. The specific formula is: , and then construct the defocus image and the actual defocus amount of the dataset.

[0076] Specifically, this method effectively utilizes simulation data and a small amount of actual data, avoids blindly training models under conditions of insufficient data, and can improve the accuracy and usability of models in scenarios with less sample data. Through data correction and refinement, the efficiency of resource utilization can be improved and the quality of training data sets can be ensured.

[0077] After obtaining the corresponding third defocus amount, the method further comprises the following steps:

[0078] Step S501, constructing a neural network model, wherein the neural network model at least includes a convolutional layer and a fully connected layer;

[0079] Step S502, training the neural network model using a training set based on a loss function, and stopping the training when the average loss value of the verification set is less than a preset value, to obtain a trained autofocus model, wherein both the training set and the verification set include data consisting of the second defocus image and the corresponding third defocus amount, and the data in the training set is different from the data in the verification set.

[0080] Specifically, taking the defocused image as input, the actual defocus amount As output, a training data set is constructed. An autofocus model including convolutional layers and fully connected layers is constructed, and the autofocus model is trained using a loss function. When the average loss value of the validation set is less than 0.01, the training is stopped to obtain the autofocus model. Then the obtained autofocus model is used to predict the actual defocus image to obtain the predicted defocus result. Based on the above method, a large amount of effective training data is obtained, and the autofocus model is obtained using these training data, which can improve the accuracy of the model. A better model can also be obtained in scenarios where samples are difficult to obtain, reducing the cost of obtaining training data.

[0081] After obtaining the trained autofocus model, the method further includes the following steps:

[0082] Step S601, obtaining a pre-prediction image and a sobel value of the pre-prediction image, and inputting the pre-prediction image into the auto-focus model to obtain a predicted defocus amount;

[0083] Step S602, adjusting the position of the photosensitive chip in the image acquisition device according to the predicted defocus amount, obtaining a predicted image using the image acquisition device, and determining the Sobel value of the predicted image;

[0084] Among them, the image acquisition device can be a lens, a camera, etc.

[0085] Step S603, when the sobel value of the image before prediction is less than the sobel value of the image after prediction, it is determined that the predicted defocus amount is accurate; when the sobel value of the image before prediction is greater than the sobel value of the image after prediction, it is determined that the predicted defocus amount is inaccurate and the predicted defocus amount is deleted.

[0086] Specifically, the prediction results of deep learning cannot be 100% accurate, so an objective method is needed to eliminate inaccurate prediction results. In order to eliminate inaccurate prediction results and improve the stability of the autofocus algorithm, a function for deleting prediction results is proposed. By comparing the sobel value of the image before prediction with the sobel value of the image after prediction, when the sobel value increases, the prediction result of the algorithm is accurate, and when the sobel value decreases, the prediction result is inaccurate and the current prediction result is eliminated.

[0087] The method for generating training data of the above-mentioned autofocus model of the present application first obtains a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocus image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image; various assembly tolerances and tolerance ranges of the image acquisition device are determined, and the wavefront distortion of the image acquisition device is simulated according to the tolerance range; multiple second defocus images are obtained based on wavefront distortion and Fraunhofer diffraction simulation, and the second defocus amount corresponding to the second defocus image is determined; each second defocus amount is corrected based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount are used for training the autofocus model. This method constructs a defocus optical model and uses the model to generate a large amount of training data with defocus information. It can achieve efficient and accurate autofocus under a small number of sample conditions, solving the problem that the autofocus method in the existing technology relies on a large amount of sample data to train the model, which is not only time-consuming but also costly.

[0088] The above-mentioned method for generating training data of the auto-focus model provides a large amount of training data, thereby solving the problem that the amount of data is too small to use a deep learning model to predict the focus position.

[0089] Example 1

[0090] like Figure 4 As shown in Table 1, a small number of real module defocus samples are collected, the defocus amount d is recorded, and the PSF radius R of the sample is calculated by calculating the second-order differential autocorrelation of the sample, and then a set of actual defocus feature data (r1, d1), (r2, d2), ..., (r 12 , d 12 ).

[0091] Table 1

[0092]

[0093] Then the actual module characteristic slope is: ;

[0094] in, ; .

[0095] like Figure 5 As shown in Table 2, by using the design values ​​of D, u, and f, the ideal imaging image of the module can be obtained through optical diffraction simulation. By changing the defocus amount in the optical diffraction simulation and calculating the PSF of the simulated defocus image, a set of theoretical defocus characteristic data (rt1, dt1), (rt2, dt2), ..., (rt 13 , dt 13 ).

[0096] Table 2

[0097]

[0098] Then the theoretical module characteristic slope is: ;

[0099] in, ; .

[0100] The entire lens is reconstructed in Zemax according to the optical lens design parameters, and the tolerance range of lens assembly is introduced so that the lens posture can change within the tolerance range. Each time it changes, the wavefront difference is calculated using Zemax and the first 15 Zernike coefficients corresponding to the wavefront difference are recorded.

[0101] Calculate the first-order correlation coefficient between each type of assembly tolerance and each Zernike coefficient, find the Zernike coefficient with a high degree of correlation with the tolerance, and calculate the fluctuation range of the corresponding Zernike coefficient. Through calculation, it is found that the absolute value of the correlation coefficient of Z4, Z5, Z6, Z7, Z8, Z11, and Z12 is greater than 0.5. It can be considered that the Zernike coefficients of these items directly affect the module assembly tolerance. The corresponding fluctuation ranges of Z4, Z5, Z6, Z7, Z8, Z11, and Z12 are (-0.02, 0.02), (-0.02, 0.02), (-0.01, 0.01), (-0.5, 0.5), (-0.5, 0.5), (-0.01, 0.01), and (-0.005, 0.005).

[0102] The type of assembly tolerance is determined according to the lens structure, and may include but is not limited to the eccentricity, tilt, air gap of the lens, the distance between each lens group and the image sensor, etc.

[0103] The correlation coefficient between each type of assembly tolerance and each Zernike term is calculated, and the Zernike terms with a high correlation with the assembly tolerance are selected so that the wavefront obtained in the subsequent simulation is closer to the real sample, thereby improving the validity of the training data and the reliability of the model.

[0104] According to the obtained related Zernike terms Z4, Z5, Z6, Z7, Z8, Z11, Z12 and their corresponding Zernike coefficient fluctuation ranges (-0.02, 0.02), (-0.02, 0.02), (-0.01, 0.01), (-0.5, 0.5), (-0.5, 0.5), (-0.01, 0.01), (-0.005, 0.005), a set of vectors containing 15 numbers is randomly generated, so that the coefficients of the Zernike terms with higher correlation change randomly within the corresponding Zernike coefficient fluctuation range, and the coefficients of the other irrelevant Zernike terms are equal to a random decimal, the absolute value of which is no more than 0.000001. In this way, the first 15 terms of the Zernike coefficients can be effectively simulated to simulate a wavefront with aberrations. , The wavefront distortion at the pupil can be described by Zernike polynomials using the following formula: ; are the coefficients of the polynomial. By accurately obtaining the coefficients of the Zernike polynomial, the optical aberration of the imaging system can be described.

[0105] Using Fraunhofer diffraction to simulate light fields at different defocus distances The image is formed on the image plane. Using the calculated G1 and G2, the defocus amount corresponding to the image is corrected , and then construct the defocus image and the actual defocus amount of the dataset.

[0106] Taking the defocused image as input, the actual defocus amount As output, a training data set is constructed. An autofocus model including convolutional layers and fully connected layers is constructed. The autofocus model is trained using a loss function. When the average loss value of the validation set is less than 0.01, the training is stopped to obtain the autofocus model. The loss function curve during the training process is shown in the figure below. Figure 6 As shown, Figure 6 TrainLoss is the training loss curve, and Validation Loss is the validation loss curve.

[0107] Then, the actual defocused image is predicted using the obtained autofocus model to obtain the predicted defocus result. The chip position is adjusted according to the predicted defocus result, and the sobel value of the image before prediction is compared with the sobel value of the image after prediction. When the sobel value increases, the prediction result of the algorithm is accurate, and when the sobel value decreases, the prediction result is inaccurate. The current prediction result is discarded and the chip returns to the position before adjustment.

[0108] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for generating training data for the autofocus model of the present application will be described in detail below in combination with specific embodiments.

[0109] This embodiment relates to a specific method for generating training data for an auto-focus model, such as Figure 7 As shown, the following steps are included:

[0110] Step 1: Collect a small amount of defocus data;

[0111] Step 2: Calculate features G1 and G2;

[0112] Step 3: Calculate the correlation coefficients between the Zernike terms and different types of tolerances and the fluctuation range of the relevant Zernike term coefficients;

[0113] Step 4: Defocus simulation generates a large amount of data and builds a data set;

[0114] Step 5: Use the dataset to train the autofocus neural network;

[0115] Step 6: Use the focus neural network to calculate the defocus amount;

[0116] Step7: Eliminate the results that failed to be predicted.

[0117] The embodiment of the present application also provides a device for generating training data of an autofocus model. It should be noted that the device for generating training data of an autofocus model in the embodiment of the present application can be used to execute the method for generating training data for an autofocus model provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been described will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0118] The following introduces a device for generating training data for an auto-focus model provided in an embodiment of the present application.

[0119] Figure 8 is a schematic diagram of a device for generating training data of an auto-focus model according to an embodiment of the present application. Figure 8 As shown, the device includes an acquisition unit 10, a determination unit 20, a simulation unit 30 and a correction unit 40, the acquisition unit 10 is used to acquire a first slope and a second slope, the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, one of the simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocused image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, one of the actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image, the actual sample image is a sample image actually collected, and the first defocused image is The defocused image is obtained by simulation; the determination unit 20 is used to determine various assembly tolerances and tolerance ranges of the image acquisition device, and simulate the wavefront distortion of the image acquisition device according to the above tolerance range; the simulation unit 30 is used to obtain multiple second defocused images based on the above wavefront distortion and Fraunhofer diffraction simulation, and determine the second defocus amount corresponding to the above second defocused image; the correction unit 40 is used to correct each of the above second defocus amounts based on the above first slope and the above second slope to obtain a corresponding third defocus amount, and the data composed of the above second defocused image and the corresponding third defocus amount are used for training the autofocus model.

[0120] The device for generating training data of the above-mentioned autofocus model of the present application includes an acquisition unit, a determination unit, a simulation unit and a correction unit, the acquisition unit is used to acquire a first slope and a second slope, the first slope is obtained by linear fitting of multiple simulated two-dimensional data, one simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocus image; the second slope is obtained by linear fitting of multiple actual two-dimensional data, one actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image; the determination unit is used to determine various assembly tolerances and tolerance ranges of the image acquisition device, and simulate the wavefront distortion of the image acquisition device according to the tolerance range; the simulation unit is used to obtain multiple second defocus images based on wavefront distortion and Fraunhofer diffraction simulation, and determine the second defocus amount corresponding to the second defocus image; the correction unit is used to correct each second defocus amount based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount are used for training the autofocus model. The device constructs a defocus optical model and uses the model to generate a large amount of training data with defocus information. It can achieve efficient and accurate autofocus under conditions of a small number of samples, solving the problem that the autofocus method in the prior art relies on a large amount of sample data to train the model, which is not only time-consuming but also costly.

[0121] In some embodiments, the acquisition unit includes a first simulation module, a first calculation module, a second calculation module, a third calculation module and a fourth calculation module, the first simulation module is used to obtain multiple first defocus images of the above-mentioned image acquisition device according to different first defocus amount simulations based on image imaging principles and optical diffraction simulations, and the above-mentioned first defocus images correspond one-to-one to the above-mentioned first defocus amounts; the first calculation module is used to respectively calculate the PSF radius of each of the above-mentioned first defocus images to obtain the simulated PSF radius corresponding to each of the above-mentioned first defocus images, and determine a group of corresponding first defocus amounts and the above-mentioned simulated PSF radius as the above-mentioned simulated two-dimensional data, and the above-mentioned first defocus image corresponds one-to-one to the above-mentioned simulated two-dimensional data; the second calculation module is used to calculate the formula according to all the above-mentioned simulated PSF radii and the first mean , calculate the first radius mean, where is the simulated PSF radius of the first defocused image, is the simulated PSF radius of the second first defocused image, is the simulated PSF radius of the mth first defocused image, is the mean value of the first radius, m is the number of the first defocused images; the third calculation module is used to calculate the formula according to all the first defocus amounts and the second mean values. , calculate and obtain the first defocus amount mean; is the first defocus amount of the first defocus image, is the first defocus amount of the second first defocus image, is the first defocus amount of the mth first defocus image, is the first defocus amount mean value; the fourth calculation module is used to calculate the first slope according to the first radius mean value, the first defocus amount mean value, all the above two-dimensional simulation data and the first slope calculation formula , the first slope is calculated, where G1 is the first slope. Through optical diffraction simulation, an ideal imaging image of the module can be obtained. By changing the defocus amount in the optical diffraction simulation and calculating the PSF of the simulated defocus image, a set of theoretical defocus feature data can be obtained, so that a large amount of effective training data can be generated in a short time.

[0122] In some embodiments, the acquisition unit includes an acquisition module, a fifth calculation module, a sixth calculation module, a seventh calculation module and an eighth calculation module, wherein the acquisition module is used to acquire multiple actual sample images and acquire the sample defocus of each of the above-mentioned actual sample images; the fifth calculation module is used to respectively calculate the PSF radius of each of the above-mentioned actual sample images, obtain the sample PSF radius corresponding to each of the above-mentioned actual sample images, and determine a group of corresponding sample defocus and sample PSF radius as the above-mentioned actual two-dimensional data, and the above-mentioned actual sample images correspond one to one with the above-mentioned actual two-dimensional data; the sixth calculation module is used to calculate the PSF radius of all the above-mentioned sample images and the third mean according to the formula , calculate the second radius mean, where is the sample PSF radius of the first actual sample image, is the sample PSF radius of the second actual sample image, is the sample PSF radius of the nth actual sample image, is the second radius mean, n is the number of the actual sample images; the seventh calculation module is used to calculate the formula according to all the sample defocus amounts and the fourth mean , calculate and obtain the second defocus amount mean; is the sample defocus amount of the first actual sample image, is the sample defocus amount of the second actual sample image, is the sample defocus amount of the nth actual sample image, is the second defocus amount mean; the eighth calculation module is used to calculate the formula according to the second radius mean, the second defocus amount mean, all the actual two-dimensional data and the second slope , the second slope is calculated, where G2 is the second slope. This will greatly improve the accuracy of the model.

[0123] In some embodiments, the determination unit includes a first processing module, a ninth calculation module, a second processing module and a second determination module, the first processing module is used to simulate the change of the above-mentioned image acquisition device within the above-mentioned tolerance range according to the above-mentioned various types of assembly tolerances and tolerance ranges, and each time the change occurs, the wavefront difference is calculated once using the optical design software and the first NZ Zernike coefficients corresponding to the current wavefront difference are recorded, where NZ is an integer greater than or equal to a preset value; the ninth calculation module is used to calculate the first-order correlation coefficient of any type of the above-mentioned assembly tolerances and any of the above-mentioned Zernike coefficients, and determine the correlation with each of the above-mentioned assembly tolerances based on all of the above-mentioned first-order correlation coefficients. The Zernike coefficient with the highest correlation is the target Zernike coefficient, and the fluctuation range of the target Zernike coefficient is determined; the second processing module is used to randomly generate multiple groups of coefficients according to all the target Zernike coefficients and the corresponding fluctuation ranges, and a group of the coefficients includes NZ vectors to obtain the coefficients of the Zernike polynomial, and the Zernike polynomial has NZ terms, wherein the target Zernike coefficient in the group of the coefficients is in the corresponding fluctuation range; the second determining ... the Zernike polynomial, the coefficients of the Zernike polynomial and the distortion formula , determine the wavefront distortion of the above image acquisition device, is the above wavefront distortion, is the above Zernike polynomial, are the coefficients of the above Zernike polynomials. A large amount of effective training data can be generated in a short time.

[0124] In some embodiments, the correction unit includes a correction module, which is used to correct the first slope, the second slope and the correction formula , each of the above second defocus amounts is corrected to obtain the corresponding third defocus amount, wherein G1 is the above first slope, G2 is the above second slope, is the third defocus amount mentioned above, This method effectively utilizes simulation data and a small amount of actual data, avoiding blindly training the model under the condition of insufficient data.

[0125] In some embodiments, there are multiple groups of training data, and the device further includes a construction module and a training module. The construction module is used to construct a neural network model after determining the second defocus image and the third defocus amount as a group of training data for the autofocus model. The neural network model at least includes a convolutional layer and a fully connected layer. The training module is used to train the neural network model using a training set based on a loss function, and stop training when the average loss value of the validation set is less than a preset value to obtain a trained autofocus model. The training set and the validation set both include data composed of the second defocus image and the corresponding third defocus amount, and the data in the training set is different from that in the validation set. The accuracy of model training is improved.

[0126] In some embodiments, the device further includes a second acquisition module, a third determination module and a fourth determination module, wherein the second acquisition module is used to acquire a pre-prediction image and a sobel value of the pre-prediction image, and input the pre-prediction image into the autofocus model to obtain a predicted defocus amount; the third determination module is used to adjust the position of the photosensitive chip in the image acquisition device according to the predicted defocus amount, and use the image acquisition device to obtain a post-prediction image, and determine the sobel value of the post-prediction image; the fourth determination module is used to determine that the predicted defocus amount is accurate when the sobel value of the pre-prediction image is less than the sobel value of the post-prediction image, and determine that the predicted defocus amount is inaccurate when the sobel value of the pre-prediction image is greater than the sobel value of the post-prediction image, and delete the predicted defocus amount. In this way, inaccurate prediction results can be eliminated, and the stability of the autofocus algorithm can be improved.

[0127] The apparatus for generating training data of the autofocus model includes a processor and a memory. The acquisition unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0128] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the auto-focus method in the prior art relies on a large amount of sample data to train the model, which is not only time-consuming but also costly.

[0129] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0130] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for generating training data of the autofocus model.

[0131] Specifically, the method for generating training data for the auto-focus model includes:

[0132] Step S201, obtaining a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one of the simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocused image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one of the actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image, wherein the actual sample image is a sample image actually collected, and the first defocused image is a defocused image obtained by simulation;

[0133] Specifically, through optical diffraction simulation, the ideal imaging image of the module can be obtained. By changing the defocus amount in the optical diffraction simulation and calculating the PSF of the simulated defocus image, a set of theoretical defocus feature data can be obtained, which can generate a large amount of effective training data in a short time. Collecting a small amount of real module defocus samples can increase the accuracy of the model.

[0134] Step S202, determining various assembly tolerances and tolerance ranges of the image acquisition device, and simulating the wavefront distortion of the image acquisition device according to the tolerance ranges;

[0135] Specifically, wavefront distortion can be described by a linear combination of Zernike polynomials, each of which corresponds to a specific type of aberration. By calculating wavefront distortion, the Zernike coefficients can be obtained, and then the main types and degrees of aberrations in the optical system can be analyzed, providing data support for system design and correction.

[0136] Step S203, obtaining a plurality of second defocused images based on the wavefront distortion and Fraunhofer diffraction simulation, and determining a second defocus amount corresponding to the second defocused images;

[0137] Specifically, by using detailed information on wavefront distortion and Fraunhofer diffraction theory, images under different defocus states can be accurately simulated. This simulation technology can generate a large number of realistic defocus images, overcoming the problems of low efficiency and slow data generation speed of traditional ray tracing and tolerance simulation.

[0138] Step S204, correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount is used for training an autofocus model.

[0139] Specifically, by correcting the defocus amount generated by the simulation based on the first slope and the second slope, the quality of training data of the autofocus model can be significantly improved, thereby improving the accuracy, generalization ability and training efficiency of the model and reducing dependence on actual samples.

[0140] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the method for generating training data of the auto-focus model when running.

[0141] Specifically, the method for generating training data for the auto-focus model includes:

[0142] Step S201, obtaining a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one of the simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocused image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one of the actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image, wherein the actual sample image is a sample image actually collected, and the first defocused image is a defocused image obtained by simulation;

[0143] Specifically, through optical diffraction simulation, the ideal imaging image of the module can be obtained. By changing the defocus amount in the optical diffraction simulation and calculating the PSF of the simulated defocus image, a set of theoretical defocus feature data can be obtained, which can generate a large amount of effective training data in a short time. Collecting a small amount of real module defocus samples can increase the accuracy of the model.

[0144] Step S202, determining various assembly tolerances and tolerance ranges of the image acquisition device, and simulating the wavefront distortion of the image acquisition device according to the tolerance ranges;

[0145] Specifically, wavefront distortion can be described by a linear combination of Zernike polynomials, each of which corresponds to a specific type of aberration. By calculating wavefront distortion, the Zernike coefficients can be obtained, and then the main types and degrees of aberrations in the optical system can be analyzed, providing data support for system design and correction.

[0146] Step S203, obtaining a plurality of second defocused images based on the wavefront distortion and Fraunhofer diffraction simulation, and determining a second defocus amount corresponding to the second defocused images;

[0147] Specifically, by using detailed information on wavefront distortion and Fraunhofer diffraction theory, images under different defocus states can be accurately simulated. This simulation technology can generate a large number of realistic defocus images, overcoming the problems of low efficiency and slow data generation speed of traditional ray tracing and tolerance simulation.

[0148] Step S204, correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount is used for training an autofocus model.

[0149] Specifically, by correcting the defocus amount generated by the simulation based on the first slope and the second slope, the quality of training data of the autofocus model can be significantly improved, thereby improving the accuracy, generalization ability and training efficiency of the model and reducing dependence on actual samples.

[0150] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0151] Step S201, obtaining a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one of the simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocused image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one of the actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image, wherein the actual sample image is a sample image actually collected, and the first defocused image is a defocused image obtained by simulation;

[0152] Step S202, determining various assembly tolerances and tolerance ranges of the image acquisition device, and simulating the wavefront distortion of the image acquisition device according to the tolerance ranges;

[0153] Step S203, obtaining a plurality of second defocused images based on the wavefront distortion and Fraunhofer diffraction simulation, and determining a second defocus amount corresponding to the second defocused images;

[0154] Step S204, correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount is used for training an autofocus model.

[0155] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0156] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0157] Step S201, obtaining a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one of the simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocused image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one of the actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image, wherein the actual sample image is a sample image actually collected, and the first defocused image is a defocused image obtained by simulation;

[0158] Step S202, determining various assembly tolerances and tolerance ranges of the image acquisition device, and simulating the wavefront distortion of the image acquisition device according to the tolerance ranges;

[0159] Step S203, obtaining a plurality of second defocused images based on the wavefront distortion and Fraunhofer diffraction simulation, and determining a second defocus amount corresponding to the second defocused images;

[0160] Step S204, correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount is used for training an autofocus model.

[0161] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0166] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0167] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0168] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0169] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0170] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0171] 1) The method for generating training data of the above-mentioned autofocus model of the present application first obtains a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocus image; the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image; various assembly tolerances and tolerance ranges of the image acquisition device are determined, and the wavefront distortion of the image acquisition device is simulated according to the tolerance range; multiple second defocus images are obtained based on wavefront distortion and Fraunhofer diffraction simulation, and the second defocus amount corresponding to the second defocus image is determined; each second defocus amount is corrected based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount are used for training the autofocus model. This method constructs a defocus optical model and uses the model to generate a large amount of training data with defocus information. It can achieve efficient and accurate autofocus under a small number of sample conditions, solving the problem that the autofocus method in the existing technology relies on a large amount of sample data to train the model, which is not only time-consuming but also costly.

[0172] 2) The device for generating training data of the above-mentioned autofocus model of the present application comprises an acquisition unit, a determination unit, a simulation unit and a correction unit, the acquisition unit is used to acquire a first slope and a second slope, the first slope is obtained by linear fitting of multiple simulated two-dimensional data, one simulated two-dimensional data comprises a simulated PSF radius and a first defocus amount of a first defocus image; the second slope is obtained by linear fitting of multiple actual two-dimensional data, one actual two-dimensional data comprises a sample PSF radius and a sample defocus amount of an actual sample image; the determination unit is used to determine various assembly tolerances and tolerance ranges of the image acquisition device, and simulate the wavefront distortion of the image acquisition device according to the tolerance range; the simulation unit is used to obtain multiple second defocus images based on wavefront distortion and Fraunhofer diffraction simulation, and determine the second defocus amount corresponding to the second defocus image; the correction unit is used to correct each second defocus amount based on the first slope and the second slope to obtain a corresponding third defocus amount, and the data consisting of the second defocus image and the corresponding third defocus amount are used for training the autofocus model. The device constructs a defocus optical model and uses the model to generate a large amount of training data with defocus information. It can achieve efficient and accurate autofocus under conditions of a small number of samples, solving the problem that the autofocus method in the prior art relies on a large amount of sample data to train the model, which is not only time-consuming but also costly.

[0173] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating training data for an auto-focus model, characterized in that: include: Acquire a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one of the simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocused image; and the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one of the actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image, wherein the actual sample image is a sample image actually collected, and the first defocused image is a defocused image obtained by simulation; Determine various assembly tolerances and tolerance ranges of the image acquisition device, and simulate the wavefront distortion of the image acquisition device according to the tolerance range; Obtaining a plurality of second defocused images based on the wavefront distortion and Fraunhofer diffraction simulation, and determining a second defocus amount corresponding to the second defocused images; Each of the second defocus amounts is corrected based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount is used for training an autofocus model.

2. The method according to claim 1, characterized in that Get the first slope, including: Based on the image imaging principle and optical diffraction simulation, a plurality of first defocus images of the image acquisition device are obtained according to simulations of different first defocus amounts, wherein the first defocus images correspond to the first defocus amounts one by one; Calculating the PSF radius of each of the first defocused images respectively to obtain a simulated PSF radius corresponding to each of the first defocused images, and determining a set of corresponding first defocus amounts and the simulated PSF radius as one of the simulated two-dimensional data, wherein the first defocused images correspond to the simulated two-dimensional data in one-to-one correspondence; According to all the simulation PSF radius and the first mean calculation formula , calculate the first radius mean, where is the simulated PSF radius of the first defocused image, is the simulated PSF radius of the second first defocused image, is the simulated PSF radius of the mth first defocused image, is the first radius mean, and m is the number of the first out-of-focus images; According to the first defocus amount and the second mean value calculation formula , calculate and obtain the first defocus amount mean; is the first defocus amount of the first first defocus image, is the first defocus amount of the second first defocus image, is the first defocus amount of the mth first defocus image, is the first defocus amount mean value; According to the first radius mean, the first defocus amount mean, all the simulation two-dimensional data and the first slope calculation formula , calculate and obtain the first slope, where G1 is the first slope.

3. The method according to claim 1, characterized in that Get the second slope, including: Collecting a plurality of actual sample images and obtaining a sample defocus value of each of the actual sample images; Calculating the PSF radius of each actual sample image respectively to obtain the sample PSF radius corresponding to each actual sample image, and determining a set of corresponding sample defocus amounts and sample PSF radii as one actual two-dimensional data, wherein the actual sample image corresponds to the actual two-dimensional data in one-to-one correspondence; According to all the sample PSF radius and the third mean calculation formula , calculate the second radius mean, where is the sample PSF radius of the first actual sample image, is the sample PSF radius of the second actual sample image, is the sample PSF radius of the nth actual sample image, is the second radius mean, and n is the number of the actual sample images; According to the defocus amount of all the samples and the fourth mean calculation formula , calculate and obtain the second defocus amount mean; is the sample defocus amount of the first actual sample image, is the sample defocus amount of the second actual sample image, is the sample defocus amount of the nth actual sample image, is the second defocus amount mean; According to the second radius mean, the second defocus amount mean, all the actual two-dimensional data and the second slope calculation formula , the second slope is calculated, where G2 is the second slope.

4. The method according to claim 1, characterized in that: Simulating and obtaining the wavefront distortion of the image acquisition device according to the tolerance range includes: According to the various assembly tolerances and tolerance ranges, the changes of the image acquisition device within the tolerance range are simulated, and each time the change occurs, the wavefront difference is calculated using the optical design software and the first NZ Zernike coefficients corresponding to the current wavefront difference are recorded, where NZ is an integer greater than or equal to a preset value; Calculate the first-order correlation coefficient between any type of the assembly tolerance and any one of the Zernike coefficients, and determine the Zernike coefficient with the highest correlation with each of the assembly tolerances as the target Zernike coefficient based on all the first-order correlation coefficients, and determine the fluctuation range of the target Zernike coefficient; According to all the target Zernike coefficients and the corresponding fluctuation ranges, a plurality of groups of coefficients are randomly generated, wherein a group of the coefficients includes NZ vectors, and coefficients of a Zernike polynomial are obtained, wherein the Zernike polynomial has NZ terms, wherein the target Zernike coefficients in a group of the coefficients are within the corresponding fluctuation range; According to the Zernike polynomial, the coefficients of the Zernike polynomial and the distortion formula , determining the wavefront distortion of the image acquisition device, is the wavefront distortion, is the Zernike polynomial, are the coefficients of the Zernike polynomial.

5. The method according to any one of claims 1 to 4, characterized in that Correcting each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount includes: Based on the first slope, the second slope and the correction formula , each of the second defocus amounts is corrected to obtain the corresponding third defocus amount, wherein G1 is the first slope, G2 is the second slope, is the third defocus amount, is the second defocus amount.

6. The method according to claim 1, characterized in that After obtaining the corresponding third defocus amount, the method further includes: Constructing a neural network model, wherein the neural network model includes at least a convolutional layer and a fully connected layer; The neural network model is trained using a training set based on a loss function, and the training is stopped when the average loss value of the validation set is less than a preset value, so as to obtain a trained autofocus model. The training set and the validation set both include data consisting of the second defocus image and the corresponding third defocus amount, and the data in the training set is different from the data in the validation set.

7. The method according to claim 6, characterized in that After obtaining the trained auto-focus model, the method further includes: Acquire a pre-prediction image and a sobel value of the pre-prediction image, and input the pre-prediction image into the auto-focus model to obtain a predicted defocus amount; Adjusting the position of a photosensitive chip in an image acquisition device according to the predicted defocus amount, obtaining a predicted image using the image acquisition device, and determining a sobel value of the predicted image; When the sobel value of the image before prediction is less than the sobel value of the image after prediction, the predicted defocus amount is determined to be accurate; when the sobel value of the image before prediction is greater than the sobel value of the image after prediction, the predicted defocus amount is determined to be inaccurate and the predicted defocus amount is deleted.

8. A device for generating training data for an auto-focus model, characterized in that: include: An acquisition unit is used to acquire a first slope and a second slope, wherein the first slope is obtained by linearly fitting a plurality of simulated two-dimensional data, wherein one of the simulated two-dimensional data includes a simulated PSF radius and a first defocus amount of a first defocused image; and the second slope is obtained by linearly fitting a plurality of actual two-dimensional data, wherein one of the actual two-dimensional data includes a sample PSF radius and a sample defocus amount of an actual sample image, wherein the actual sample image is a sample image actually collected, and the first defocused image is a defocused image obtained by simulation; A determination unit, used to determine various assembly tolerances and tolerance ranges of the image acquisition device, and simulate the wavefront distortion of the image acquisition device according to the tolerance range; a simulation unit, configured to obtain a plurality of second defocused images based on the wavefront distortion and Fraunhofer diffraction simulation, and determine a second defocus amount corresponding to the second defocused images; A correction unit is used to correct each of the second defocus amounts based on the first slope and the second slope to obtain a corresponding third defocus amount, and data consisting of the second defocus image and the corresponding third defocus amount is used for training an autofocus model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for generating training data for the auto-focus model according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for generating training data for executing the autofocus model described in any one of claims 1 to 7.

Images