Image automatic focusing method, device, storage medium and electronic device

Abstract

The present application provides an automatic focusing method, device, storage medium and electronic device for an image, the method comprising: cutting out a knife edge image from a target image, determining feature data of the target image based on the knife edge image, the feature data comprising at least one of the following: the ESF radius, PSF radius and extreme radius of the target image, the knife edge image being an image including a clear blade edge; using multiple sets of training data to obtain an overfocus and underfocus classification neural network based on FocusNet network training; inputting the target image into the overfocus and underfocus classification neural network to obtain a target defocus direction of the target image; determining the defocus amount of the target image based on the feature data of the target image and the target defocus direction, and automatically focusing the target image based on the defocus amount. The method uses three feature indicators that are linearly related to the defocus amount, and reversely derives the defocus amount through a linear model or a high-order model to quickly predict the quasi-focus position of the chip.

Description

Technical Field

[0001] The present application relates to the field of autofocus technology, and in particular to an image autofocus method, an image autofocus device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Image clarity is one of the important indicators of the quality of a lens, and MTF is the most important indicator of lens image clarity. In actual production, multiple tolerances of multiple lenses and the eccentricity of lens assembly are combined together, making the actual MTF curve complicated and the MTF yield difficult to guarantee. The MTF-Through-Focus function is of great significance and role in analyzing the overall performance of the module and timely discovering the optical design defects of the module. Optical engineers can use through focus data to discover optical problems in lens monomers or module assembly and make adjustments. At this stage, MTF-Through-Focus is mainly obtained by changing the axial relative position of the lens and the chip during the test, recording multiple frames of defocus images and testing the MTF values ​​of each field of view at the current position, and finally obtaining the defocus curve of the MTF value changing with the axial distance. Since the entire process requires multiple measurements, the efficiency is low. Summary of the invention

[0003] The main purpose of the present application is to provide an automatic focusing method for an image, an automatic focusing device for an image, a computer-readable storage medium and an electronic device, so as to at least solve the problem of low efficiency and large amount of calculation in calculating the focus position of an image in the prior art.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, there is provided an automatic focusing method for an image, comprising: acquiring a target image, and cropping a knife edge image from the target image, determining feature data of the target image according to the knife edge image, the feature data comprising at least one of the following: an ESF radius, a PSF radius and an extreme radius of the target image, the knife edge image being an image including a clear knife edge, and the knife edge being a straight line boundary between two adjacent areas whose brightness difference is greater than a preset difference; using multiple sets of training data to obtain an overfocus or underfocus classification neural network based on FocusNet network training, each set of training data in the multiple sets of training data comprises: a historical image and a historical defocus direction corresponding to the historical image acquired within a historical time period; inputting the target image into the overfocus or underfocus classification neural network to obtain a target defocus direction of the target image; determining a defocus amount of the target image according to the feature data of the target image and the target defocus direction, and automatically focusing the target image according to the defocus amount.

[0005] Optionally, the feature data includes the ESF radius of the target image, and the feature data of the target image is determined according to the knife edge image, including: graying and removing noise from the knife edge image to obtain a first processed image; using an edge detection algorithm to determine an initial knife edge in the first processed image, and fitting the initial knife edge to obtain a straight knife edge by least squares method, wherein the edge detection algorithm is a Laplace algorithm or a Canny edge detector algorithm; using a difference algorithm to perform sub-pixel edge positioning on the straight knife edge, and projecting along the knife edge direction of the straight knife edge to obtain a discrete knife edge brightness distribution curve; sampling the knife edge brightness distribution curve at equal intervals to obtain sampling point data, and fitting the sampling point data with a fitting function to obtain an edge extension function curve, wherein the fitting function is a cubic spline interpolation function or a Fermi function; determining the ESF radius of the target image according to the edge extension function curve.

[0006] Optionally, determining the ESF radius of the target image according to the edge spread function curve includes: calculating the ESF radius of the target image according to the maximum value m of the edge spread function curve, the minimum value n of the edge spread function curve, and a first threshold calculation formula: And the second threshold calculation formula , respectively determine the first threshold and the second threshold, wherein, is the first threshold, is the second threshold, 0≤a≤1; determining the point in the edge spread function curve that is closest to the first threshold as the first data point, and determining the point in the edge spread function curve that is closest to the second threshold as the second data point; obtaining the distance calculation formula , and calculate the absolute distance between the first data point and the second data point according to the distance calculation formula to obtain the ESF radius, is the ESF radius, is the first data point, is the second data point.

[0007] Optionally, the feature data includes a PSF radius of the target image, and determining the feature data of the target image according to the knife-edge image includes: converting the knife-edge image into a grayscale image; performing second-order differential processing on the grayscale image using a Laplace operator to calculate an autocorrelation matrix of the grayscale image to obtain a PSF image, and extracting center data from the autocorrelation matrix, wherein the center data includes a plurality of center sub-data, and the center data is center row data or center column data; using a wavelet transform formula , perform wavelet transform on each of the central sub-data to obtain the corresponding transformed data, where, is the transformed data after c has been subjected to the wavelet transform, is the wavelet basis used in the wavelet transform, W is the wavelet transform, and c is the central sub-data; threshold processing is performed on all the transformed data, and all the transformed data greater than 0 and the transformed data with an absolute value less than a preset value are determined as 0 to obtain processed data; the position of the kth minimum point and the position of the k+1th minimum point in the processed data are determined as the first position point and the second position point respectively; according to the radius calculation formula and the position difference between the first position point and the second position point, to determine the PSF radius, wherein, is the PSF radius, is the position of the first position point, is the position of the second position point.

[0008] Optionally, the feature data includes the maximum radius of the target image, and determining the feature data of the target image according to the knife-edge image includes: according to the maximum brightness value of the knife-edge image and the minimum brightness value of the knife-edge image, a third threshold calculation formula and the fourth threshold budget formula , respectively determine the third threshold value and the fourth threshold value, is the third threshold, is the fourth threshold, is the maximum brightness of the knife-edge image, is the minimum brightness value of the knife-edge image, 0≤b≤1; calculate the absolute value of the difference between each pixel point in the knife-edge image and the third threshold to obtain the corresponding first absolute value, and calculate the absolute value of the difference between each pixel point in the knife-edge image and the fourth threshold to obtain the corresponding second absolute value; determine the maximum radius of the target image based on all the first absolute values ​​and all the second absolute values.

[0009] Optionally, determining the maximum radius of the target image according to all the first absolute values ​​and all the second absolute values ​​includes: arranging all the first absolute values ​​from small to large, and determining the pixel points corresponding to the first absolute values ​​preset as a first group of feature points; arranging all the second absolute values ​​from small to large, and determining the pixel points corresponding to the second absolute values ​​preset as a second group of feature points; calculating the distance between any two feature points in the first group of feature points and the second group of feature points to obtain multiple calculated distances; calculating the maximum radius according to the formula , determine the maximum radius of the target image, where, is the maximum radius, is the coordinate of the i-th point in the first set of feature points, is the coordinate of the jth point in the second set of feature points.

[0010] Optionally, determining the defocus amount of the target image according to the feature data of the target image and the target defocus direction includes: obtaining a relationship model , the relationship model is a relationship model between the feature data of the target image and the defocus position of the target image, wherein, is the ESF radius, is the PSF radius, is the maximum radius, is the defocus position of the target image, a is the first coefficient, b is the second coefficient, c is the third coefficient, and d is the fourth coefficient, wherein d is not zero, and at least one of a, b, and c is not zero; according to the relationship model, the defocus position of the target image is determined; and a defocus amount calculation formula is obtained. ,in, is the target image, is the target defocus direction, is the defocus position of the target image, is the defocus amount of the target image; and the defocus amount of the target image is determined according to the defocus amount calculation formula.

[0011] According to another aspect of the present application, an automatic focusing device for an image is provided, comprising: an acquisition unit, for acquiring a target image, and cutting out a knife-edge image from the target image, and determining feature data of the target image according to the knife-edge image, wherein the feature data comprises at least one of the following: an ESF radius, a PSF radius and an extreme radius of the target image, wherein the knife-edge image is an image including a clear knife edge, and the knife edge is a straight line boundary between two adjacent areas whose brightness difference is greater than a preset difference; a training unit, for obtaining an overfocus or underfocus classification neural network based on FocusNet network training using multiple sets of training data, wherein each set of training data in the multiple sets of training data comprises: a historical image and a historical defocus direction corresponding to the historical image, obtained within a historical time period; a first processing unit, for inputting the target image into the overfocus or underfocus classification neural network to obtain a target defocus direction of the target image; and a second processing unit, for determining a defocus amount of the target image according to the feature data of the target image and the target defocus direction, and automatically focusing the target image according to the defocus amount.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for automatically focusing an image.

[0013] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the above-described image autofocus methods.

[0014] Applying the technical solution of the present application, the above-mentioned image autofocus method first cuts out the blade image from the target image, determines the characteristic data of the target image based on the blade image, and the characteristic data includes at least one of the following: the ESF radius, PSF radius and maximum radius of the target image, and the blade image is an image including a clear blade edge; then multiple sets of training data are used to obtain an overfocus and underfocus classification neural network based on FocusNet network training; then the target image is input into the overfocus and underfocus classification neural network to obtain the target defocus direction of the target image; finally, the defocus amount of the target image is determined according to the characteristic data of the target image and the target defocus direction, and the target image is automatically focused according to the defocus amount. This method uses three characteristic indicators that are linearly related to the defocus amount, and infers the defocus amount through a linear model or a high-order model. Without changing the existing configuration, a single picture is used to quickly predict the quasi-focus position of the chip, reduce time cost, and solve the problem of low efficiency and large amount of calculation in calculating the focus position of the image in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A schematic diagram of a process of an automatic focusing method of an image provided according to an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram of an ESF radius algorithm provided according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram of a filtering effect of a wavelet transform provided according to an embodiment of the present application is shown;

[0019] Figure 4 A schematic diagram showing a method for calculating a maximum radius provided according to an embodiment of the present application is shown;

[0020] Figure 5 A schematic diagram showing another method for calculating the maximum radius provided according to an embodiment of the present application is shown;

[0021] Figure 6 A scatter plot of ESF radius and defocus amount provided according to an embodiment of the present application is shown;

[0022] Figure 7 A scatter plot of a maximum radius and defocus amount provided according to an embodiment of the present application is shown;

[0023] Figure 8 A schematic diagram of an image acquisition device and a preprocessing process in an embodiment provided according to an embodiment of the present application is shown;

[0024] Fig. 9 A schematic diagram of a process flow of another image auto-focusing method provided according to an embodiment of the present application is shown;

[0025] Fig.10 A structural block diagram of an automatic focusing device for an image provided according to an embodiment of the present application is shown.

[0026] The above drawings include the following reference numerals:

[0027] 01. Image acquisition system; 011. Target; 012. Optical module; 013. Photosensitive chip; 02. Defocus map; 03. Central field of view test area; 04. First knife edge image; 05. Second knife edge image; 06. Third knife edge image; 07. Fourth knife edge image. DETAILED DESCRIPTION

[0028] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] As introduced in the background technology, in the prior art, the imaging quality index of the lens at each position can be calculated by stepping through the actual position of the chip. The commonly used index is the MTF value. The relationship between this index and the focus position is nonlinear, and when the defocus amount is large, the MTF does not change with the change of the defocus amount. In order to solve the problem of low efficiency and large amount of calculation in calculating the focus position of the image in the prior art, the embodiments of the present application provide an image auto-focus method, an image auto-focus device, a computer-readable storage medium and an electronic device.

[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0033] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device.

[0034] In this embodiment, a method for automatically focusing an image running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] Figure 1 FIG. 1 is a flow chart of an automatic focusing method for an image according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0036] Step S101, acquiring a target image, and cutting out a knife edge image from the target image, determining feature data of the target image according to the knife edge image, wherein the feature data includes at least one of the following: an ESF radius, a PSF radius, and an extreme radius of the target image, wherein the knife edge image is an image including a clear knife edge, and the knife edge is a straight line boundary between two adjacent areas whose brightness difference is greater than a preset difference;

[0037] Specifically, by finding a linear clarity index associated with the defocus distance, the focus position can be quickly and accurately calculated in a single image, thereby improving production efficiency.

[0038] The characteristic data includes the ESF radius of the target image, and determining the characteristic data of the target image according to the knife edge image includes the following steps:

[0039] Step S201, graying and removing noise from the knife edge image to obtain a first processed image;

[0040] Step S202, using an edge detection algorithm to determine an initial edge in the first processed image, and fitting the initial edge to obtain a straight edge by a least square method, wherein the edge detection algorithm is a Laplace algorithm or a Canny edge detector algorithm;

[0041] Step S203, using a difference algorithm to perform sub-pixel edge positioning on the straight edge, and projecting along the edge direction of the straight edge to obtain a discrete edge brightness distribution curve;

[0042] Step S304, sampling the blade edge brightness distribution curve at equal intervals to obtain sampling point data, and fitting the sampling point data with a fitting function to obtain an edge extension function curve, wherein the fitting function is a cubic spline interpolation function or a Fermi function;

[0043] Step S205: determining the ESF radius of the target image according to the edge spread function curve.

[0044] Specifically, the ESF curve can be selected as an image containing a clear edge, which should be a straight boundary between two adjacent and high-contrast uniformly bright and dark objects. Subsequently, the image is grayed to ensure that the processed data is one-dimensional brightness information and remove image noise, thereby improving the signal-to-noise ratio and reducing the error of subsequent calculations. Next, an edge detection algorithm (such as the Laplace operator or the Canny edge detector) is used to accurately find the edge position in the image, and a more accurate straight edge is obtained by least squares fitting. In order to obtain more accurate edge position information, sub-pixel edge positioning is also required on the fitted straight edge, which is usually achieved by an interpolation algorithm. Thereafter, a discrete edge brightness distribution curve is obtained by projection along the edge direction, and the curve is sampled at equal intervals to obtain sufficient sampling point data. Finally, these sampled data are fitted using an appropriate function (such as cubic spline interpolation or Fermi function) to obtain a smooth edge spread function (ESF) curve.

[0045] In the above example, the Laplacian operator can be, ‌Laplacian operator‌ is a second-order derivative operator used for edge detection, which detects edges by calculating the second-order derivative of the image. The Laplacian operator performs a second convolution on the image to find the rate of change of the pixel value, thereby enhancing the edge contour. However, it does not provide directional information of the edge, so it is usually necessary to combine it with other methods, such as the Sobel operator, to obtain more accurate edge information. The Laplacian operator is sensitive to noise, so in the application, it may be necessary to smooth the image first to reduce the impact of noise. In image processing, edge detection is an important step for identifying the boundaries and contours of objects in the image. In addition to the Laplacian operator, there are other commonly used edge detection operators, such as the Sobel operator and the Canny edge detector. The Sobel operator is a first-order derivative operator that is used to calculate an approximate value of the brightness of the image and detects edges by combining Gaussian smoothing and differential derivation. It has a certain smoothing effect on noise, provides more accurate edge direction information, and performs well in images with more noise. Canny edge detection is a more complex multi-step algorithm with good noise suppression and edge connection capabilities, and is often used in applications that require high-precision edge detection. In general, the choice of which edge detection method to use depends on the specific application requirements, as well as the requirements for noise and accuracy. Laplace operator, Sobel operator and Canny edge detection each have their own characteristics and are suitable for different scenarios. In practical applications, you may choose the appropriate method based on specific needs or combine them to achieve the best results.

[0046] In digital image processing, the discrete Laplace transform of an image is usually implemented through a convolution operation. The discrete Laplace transform can be viewed as applying a specific convolution kernel (also called a filter) to each pixel in the image. Assume that I(x,y) is the input image, L(x,y) is the output image, and L is the selected Laplace operator convolution kernel. The discrete Laplace transform can be expressed through the convolution operation as: .

[0047] Wherein, determining the ESF radius of the target image according to the edge spread function curve comprises the following steps:

[0048] Step S2051: Calculate the value of the edge spread function curve according to the maximum value m, the minimum value n, and the first threshold value. And the second threshold calculation formula , respectively determine the first threshold and the second threshold, wherein, is the first threshold, is the second threshold, 0≤a≤1;

[0049] Step S2052, determining the point in the edge spread function curve that is closest to the first threshold as a first data point, and determining the point in the edge spread function curve that is closest to the second threshold as a second data point;

[0050] Step S2053, obtain the distance calculation formula , and calculate the absolute distance between the first data point and the second data point according to the distance calculation formula to obtain the ESF radius, is the ESF radius, is the first data point, is the second data point.

[0051] Specifically, Figure 2 It is a schematic diagram of the ESF radius algorithm. Figure 2 As shown, the blue curve is the ESF curve, the red line segment is the characteristic radius esf_radiu calculated according to the above algorithm, a=0.86, the horizontal axis represents the position along the edge direction, in pixels (pixel). The vertical axis represents the light intensity or grayscale value, which is usually a dimensionless relative value. For 8-bit images, the range is from 0 to 255, where 0 represents complete darkness and 255 represents maximum brightness or the brightest grayscale value.

[0052] In image processing, point spread function (PSF) is an important concept that describes how a point light source in an imaging system spreads into an image. The radius of the PSF is a key parameter for evaluating image quality and performing system correction. Traditional methods are often complex and computationally intensive when calculating the PSF radius, while the embodiments of the present application provide a more efficient and accurate method for calculating the PSF radius. The specific steps are as follows.

[0053] The feature data includes the PSF radius of the target image, and determining the feature data of the target image according to the knife-edge image includes the following steps:

[0054] Step S301, converting the knife edge image into a grayscale image;

[0055] Step S302, performing second-order differential processing on the grayscale image using a Laplace operator to calculate an autocorrelation matrix of the grayscale image to obtain a PSF image, and extracting center data from the autocorrelation matrix, wherein the center data includes a plurality of center sub-data, and the center data is center row data or center column data;

[0056] Step S303, using the wavelet transform formula , perform wavelet transform on each of the central sub-data to obtain the corresponding transformed data, where, is the transformed data after c has been subjected to the wavelet transform, is the wavelet basis used in the wavelet transform, W is the wavelet transform, and c is the central sub-data;

[0057] Step S304, performing threshold processing on all the transformed data, determining all the transformed data greater than 0 and the transformed data with an absolute value less than a preset value as 0, and obtaining processed data;

[0058] Step S305, determining the position of the kth minimum point and the position of the k+1th minimum point in the processed data as the first position point and the second position point respectively; wherein k=2;

[0059] Step S306, according to the radius calculation formula and the position difference between the first position point and the second position point, to determine the PSF radius, wherein, is the PSF radius, is the position of the first position point, is the position of the second position point.

[0060] Specifically, in the prior art, the focus position can be predicted by calculating the PSF radius, but this embodiment makes further improvements. The specific improvement is to add wavelet transform filtering.

[0061] like Figure 3 As shown in the figure, if the input image is in color, it is converted to a grayscale image image. The image is processed by the second-order differential of the Laplace operator laplace. The PSF image is obtained by calculating the autocorrelation between the processed image and itself, psf_image = corr(laplace(image)).

[0062] Extract the center row or center column data c from the autocorrelation matrix. Perform wavelet transform on the extracted data. In this embodiment, Next, threshold processing is performed on the transformed coefficients: first, all coefficients greater than 0 are set to 0; then, coefficients with absolute values ​​less than 0.5 are also set to 0.

[0063] The characteristic data includes the maximum radius of the target image, and determining the characteristic data of the target image according to the knife edge image includes the following steps:

[0064] Step S401: According to the maximum brightness value of the knife-edge image and the minimum brightness value of the knife-edge image, a third threshold calculation formula is and the fourth threshold budget formula , respectively determine the third threshold value and the fourth threshold value, is the third threshold, is the fourth threshold, is the maximum brightness of the knife-edge image, is the minimum brightness value of the knife edge image, 0≤b≤1;

[0065] Specifically, Figure 4 As shown, the above steps can accurately find two groups of specific feature points in the image and calculate the minimum distance between them. The third threshold and the fourth threshold correspond to specific proportional positions of the ESF curve, where b is 86% and 1-b=14%. The two ratios of 86% and 14% are arbitrary. The only requirement is that the sum of the two ratios is equal to 1. After it is equal to 1, the heights of the two green areas in the figure are consistent.

[0066] In particular, in this embodiment, the 14% and 86% have special meanings in optics, indicating that the energy of the light spot will be concentrated in this area. For example, the distance between 14% and the boundary of pure black is the same as the distance between 86% and the boundary of pure white. The value of 1 is equal to 1 in physics to ensure symmetry, and in mathematics to ensure that the sum of these two values ​​is equal to mn.

[0067] Step S402, calculating the absolute value of the difference between each pixel in the knife-edge image and the third threshold to obtain a corresponding first absolute value, and calculating the absolute value of the difference between each pixel in the knife-edge image and the fourth threshold to obtain a corresponding second absolute value;

[0068] Step S403: determining the maximum radius of the target image according to all the first absolute values ​​and all the second absolute values.

[0069] Wherein, determining the maximum radius of the target image according to all the first absolute values ​​and all the second absolute values ​​comprises the following steps:

[0070] Step S4031, arranging all the first absolute values ​​from small to large, and determining the pixel points corresponding to the first preset absolute values ​​as a first group of feature points;

[0071] Step S4032, arranging all the second absolute values ​​from small to large, and determining the pixel points corresponding to the first preset second absolute values ​​as the second group of feature points;

[0072] Step S4033, calculating the distance between any two feature points in the first group of feature points and the second group of feature points to obtain a plurality of calculated distances;

[0073] Step S4034, according to the maximum radius calculation formula , determine the maximum radius of the target image, where, is the maximum radius, is the coordinate of the i-th point in the first set of feature points, is the coordinate of the jth point in the second set of feature points.

[0074] Specifically, calculate the absolute value of the difference between each pixel value of the knife edge image and the thresholds value1 and value2. Select the N (e.g., N=6) pixels closest to value1 and value2 as feature points. Calculate all distances between the first set of feature points (points close to value1) and the second set of feature points (points close to value2). Find the minimum distance among these distances, which is the desired maximum radius. Figure 5 As shown, Figure 5 The red circles are the first set of feature points found, the green circles are the second set of feature points found, and the blue lines are the first six shortest line segments found (there is actually only one shortest line segment, and the top six shortest line segments are drawn here).

[0075] Step S202, using multiple sets of training data to obtain an over-focus and under-focus classification neural network based on FocusNet network training, wherein each set of training data in the multiple sets of training data includes: historical images and historical defocus directions corresponding to the historical images acquired in a historical time period;

[0076] Specifically, the above steps achieve accurate prediction of the camera autofocus position by extracting and processing features of the image, combining polynomial fitting technology and deep learning technology.

[0077] Collect a large amount of image data and record the chip's defocus distance when collecting images, crop the image of a specific field of view from the original image, and then crop the knife edge from the image. One or more feature values ​​(such as ESF radius, PSF radius, and maximum radius) calculated from the knife edge image are used as input data, and the absolute value of the defocus amount is used as output data to construct the focus depth data set of the subsequent autofocus position prediction model.

[0078] Step S203, inputting the target image into the over-focus / under-focus classification neural network to obtain a target defocus direction of the target image;

[0079] Among them, the over-focus and under-focus classification neural network FocusNet is trained so that the network can judge whether the current chip position is in an over-focus state or an under-focus state based on the input image. By taking the image of a specific field of view as input and the sign of the defocus amount as a label, that is, the output target defocus direction is a sign, such as a negative sign for under-focus and a non-negative sign for over-focus (or a negative sign for over-focus and a non-negative sign for under-focus), a focus direction dataset is constructed.

[0080] Step S204 , determining a defocus amount of the target image according to the feature data of the target image and the target defocus direction, and automatically focusing the target image according to the defocus amount.

[0081] Wherein, determining the defocus amount of the target image according to the characteristic data of the target image and the target defocus direction comprises the following steps:

[0082] Step S2041, obtaining a relationship model , the relationship model is a relationship model between the feature data of the target image and the defocus position of the target image, wherein, is the ESF radius, is the PSF radius, is the maximum radius, is the defocus position of the target image, a is the first coefficient, b is the second coefficient, c is the third coefficient, and d is the fourth coefficient, wherein d is not zero, and at least one of a, b, and c is not zero;

[0083] Step S2042, determining the defocus position of the target image according to the relationship model;

[0084] Step S2043, obtaining a defocus calculation formula ,in, is the target image, is the target defocus direction, is the defocus position of the target image, is the defocus amount of the target image;

[0085] Step S2044: determining the defocus amount of the target image according to the defocus amount calculation formula.

[0086] Specifically, a, b, and c are not all equal to 0. When a, b, and c are not equal to 0, the method calculates the defocus amount through three features. When only any one of a, b, and c is equal to 0, the method calculates the defocus amount through any two features. When only any one of a, b, and c is not equal to 0, the method calculates the defocus amount through any one feature.

[0087] For higher order polynomials, such as second order, we can generalize this model. In the second order case, in addition to the first order terms, we also include the square terms of each characteristic variable. The general formula of the second order polynomial can be written as: ;

[0088] in are all undetermined coefficients, and at least one of them is The non-zero coefficients outside.

[0089] For higher order polynomials, we can continue to add higher power terms and higher order cross terms. The general form of an n-th order polynomial can be written as: .

[0090] in ,in, are undetermined coefficients, at least one of which is External non-zero coefficients.

[0091] Among them, when the accuracy of the calculation result does not meet the production requirements, a higher-order polynomial can be used.

[0092] like Figure 6 and Figure 7 As shown, Figure 6 It is a scatter plot of ESF radius and defocus (here defocus refers to the focus position in the figure). Figure 7 is a scatter plot of the maximum radius and defocus amount, from Figure 6 It can be seen that the ESF radius is linearly related to the defocus amount. Figure 7 It can be seen that the maximum radius and the defocus amount are also linearly related. The defocus amount here refers to the theoretical imaging position of the lens. When the chip is located at this theoretical imaging position, the imaging quality is the highest. When the chip deviates from the theoretical imaging position of the lens, that is, when the actual axial position of the chip is offset relative to the theoretical imaging position, the imaging quality will decrease. The defocus amount is a measure of the degree of offset between the actual imaging position of the chip and the theoretical imaging position. The reference point of the focus position is the theoretical imaging position of the lens. The defocus amount is defined as the offset between the actual position of the photosensitive chip and the theoretical imaging position. In the field of mobile phone module assembly, this offset is usually in the micron range. The position here is the axial position. The x-direction offset and y-direction offset perpendicular to the axial position plane are not considered. Only the axial offset is calculated. The axial offset here is the defocus amount mentioned many times in this briefing book. The focus position is essentially the defocus amount, but the focus position is relative to the current position of the chip. Set the current axial position of the chip as the starting point, then the focus position of the chip is the theoretical imaging position of the lens, and the distance between the chip focus position and the current actual position of the chip is the defocus amount.

[0093] In the past, many clarity evaluation indicators, such as MTF, SFR, Sobel, etc., have a nonlinear relationship with the defocus amount. Linear relationship is particularly important for autofocus algorithms, so the focus of this patent is to protect the three linear feature quantities of ESF radius, PSF radius, and maximum radius. This is why first-order polynomial fitting can be used in this step.

[0094] The three linear characteristic quantities, ESF radius, PSF radius, and maximum radius, can be used to characterize the imaging quality, and the relationship between these three indicators and the defocus amount is linear, which means that the defocus amount can be inferred by calculating any one of the indicators through a linear model or a high-order model, that is, the distance of the actual position of the current chip offset from the theoretical imaging position, that is, the focus position can be calculated.

[0095] and Figure 6 and Figure 7 The points on the graph come from different samples of the same model, but the overall slope of the scatter plot composed of these points is consistent, which also provides a guarantee for single-image prediction. In the past, many autofocus algorithms needed to collect at least two images to calculate the slope, but the slope of the clarity and defocus amount of the present invention is consistent, so the coefficient (the defocus position of the target image) can be calculated first, and then the defocus amount can be calculated using only a single image.

[0096] The automatic focusing method of the above-mentioned image of the present application firstly cuts out the blade image from the target image, determines the characteristic data of the target image according to the blade image, and the characteristic data includes at least one of the following: the ESF radius, PSF radius and maximum radius of the target image, and the blade image is an image including a clear blade edge; then, multiple sets of training data are used to obtain an overfocus and underfocus classification neural network based on FocusNet network training; then, the target image is input into the overfocus and underfocus classification neural network to obtain the target defocus direction of the target image; finally, the defocus amount of the target image is determined according to the characteristic data of the target image and the target defocus direction, and the target image is automatically focused according to the defocus amount. The method uses three characteristic indicators that are linearly related to the defocus amount, and infers the defocus amount through a linear model or a high-order model. Without changing the existing configuration, a single picture is used to quickly predict the quasi-focus position of the chip, reduce time cost, and solve the problem of low efficiency and large amount of calculation in calculating the focus position of the image in the prior art.

[0097] Among them, autofocus technology involves components such as light source, target, image acquisition system, optical module and testing software.

[0098] Example

[0099] like Figure 8As shown, the image acquisition system 01 is composed of a target 011, an optical module 012 and a photosensitive chip 013. The distance from the photosensitive chip 013 to the back focal plane of the optical module 012 is called the defocus amount. By adjusting the axial relative position between the optical module 012 and the photosensitive chip 013, a series of images 02 {IMG1, IMG2, IMG3, ..., IMGN} in different defocus states can be recorded, and the modulation transfer function (MTF) value {MTF1, MTF2, MTF3, ..., MTFN} of each field of view at each position and its corresponding chip position {l1, l2, l3, ..., lN} are measured at the same time, thereby obtaining a defocus curve of the MTF value varying with the axial distance. The peak position lx (where 1≤x≤N) of the MTF value on the curve is the back focal plane position of the optical module 012. Therefore, for each defocus image 02, its defocus distance can be expressed as {l1-lx, l2-lx, l3-lx, ..., lN-lx}. Then, select other modules of the same model and repeat the above process to collect 10,000 sets of defocus image-defocus pairing data to form the original autofocus data set.

[0100] Based on this original data set, a positioning algorithm is used to intercept the central field of view test area 03 of each defocus image 02, and the first knife edge image 04 is further extracted from the test area. By analyzing the first knife edge image 04, the edge spread function (ESF) curve can be obtained and standardized to between 0 and 1. According to the maximum value 1 and the minimum value 0 of the ESF curve, two thresholds value1=0.86 and value2=0.14 are set. The standardized ESF data is traversed to determine the positions loc1 and loc2 of the data points closest to the two thresholds, and then the absolute distance between the two points is calculated, which is defined as the characteristic radius esf_radiu= |loc1 - loc2|. In this way, 10,000 characteristic radii esf_radiu can be calculated from 10,000 defocus images, which match the corresponding defocus values ​​in the original data set and jointly construct the focus depth data set.

[0101] In order to establish a focus depth prediction model, the least squares method is used to solve the equation Parameters a and d in . In addition, the focus direction dataset was constructed by taking the central field of view test area 03 from the 10,000 defocus images in the original autofocus dataset and combining the sign of the defocus amount (negative sign represents underfocus, non-negative sign represents overfocus). This dataset was used to train a neural network model based on the MobileNetV2 architecture, using the 2 norm as the loss function, the Adam optimizer, the initial learning rate of 0.0003, and the training cycle of 1000 rounds. Whenever the model accuracy improves, the updated model is saved as FocusNet.

[0102] In practical applications, for the module to be inspected, the defocus map 02 is obtained through the image acquisition system 01, and then two knife edge images in the horizontal direction (the first knife edge image 04 and the second knife edge image 05) and two knife edge images in the vertical direction (the third knife edge image 06 and the fourth knife edge image 07) are extracted. The ESF curve is calculated and standardized from these four knife edge images respectively, and the corresponding data point positions are found based on the same thresholds value1 and value2, and the characteristic radius esf_radiu is calculated. According to the previously established model, the horizontal focus depth |distance_S| and the vertical focus depth |distance_T| can be calculated. The defocus map 02 is input into the trained FocusNet model to predict the defocus direction direction. Taking into account the focus depth |distance_S|, |distance_T| and the defocus direction direction, the precise focusing position of the photosensitive chip is finally determined. .

[0103] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the image auto-focusing method of the present application will be described in detail below in conjunction with specific embodiments.

[0104] This embodiment relates to a specific automatic focusing method for an image, such as Fig. 9 As shown, the following steps are included:

[0105] Step S1: build a measurement optical path;

[0106] Step S2: Collecting a large amount of image format data and recording the chip when collecting the image;

[0107] Step S3: constructing a focus depth data set and a focus direction data set;

[0108] Step S4: Solving using the focus depth dataset The coefficients in ;

[0109] Step S5: using the focus direction data set to train an over-focus and under-focus classification neural network FocusNet;

[0110] Step S6: Calculate the chip focal position .

[0111] The embodiment of the present application also provides an automatic focusing device for an image. It should be noted that the automatic focusing device for an image in the embodiment of the present application can be used to execute the automatic focusing method for an image provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0112] The following is an introduction to the automatic focusing device for an image provided in an embodiment of the present application.

[0113] Fig.10 is a schematic diagram of an automatic focusing device for an image according to an embodiment of the present application. Fig.10 As shown, the device includes an acquisition unit 10, a training unit 20, a first processing unit 30 and a second processing unit 40. The acquisition unit 10 is used to acquire a target image and cut out a knife edge image from the target image, and determine feature data of the target image according to the knife edge image, wherein the feature data includes at least one of the following: an ESF radius, a PSF radius and an extreme radius of the target image, wherein the knife edge image is an image including a clear knife edge, and the knife edge is a straight line boundary between two adjacent areas whose brightness difference is greater than a preset difference; the training unit 20 is used to obtain an overfocus and underfocus classification neural network based on FocusNet network training using multiple sets of training data, wherein each set of training data in the multiple sets of training data includes: a historical image and a historical defocus direction corresponding to the historical image obtained in a historical time period; the first processing unit 30 is used to input the target image into the overfocus and underfocus classification neural network to obtain a target defocus direction of the target image; the second processing unit 40 is used to determine the defocus amount of the target image according to the feature data of the target image and the target defocus direction, and automatically focus the target image according to the defocus amount.

[0114] The above-mentioned image auto-focus device of the present application includes an acquisition unit, a training unit, a first processing unit and a second processing unit. The acquisition unit is used to cut out a knife edge image from a target image, and determine the characteristic data of the target image according to the knife edge image. The characteristic data includes at least one of the following: the ESF radius, the PSF radius and the maximum radius of the target image, and the knife edge image is an image including a clear blade edge; the training unit is used to obtain an over-focus and under-focus classification neural network based on FocusNet network training using multiple sets of training data; the first processing unit is used to input the target image into the over-focus and under-focus classification neural network to obtain the target defocus direction of the target image; the second processing unit is used to determine the defocus amount of the target image according to the characteristic data of the target image and the target defocus direction, and automatically focus the target image according to the defocus amount. The device uses three characteristic indicators that are linearly related to the defocus amount, and inverts the defocus amount through a linear model or a high-order model. Without changing the existing configuration, a single picture is used to quickly predict the quasi-focus position of the chip, reduce time cost, and solve the problem of low efficiency and large amount of calculation in calculating the focus position of the image in the prior art.

[0115] In some instances, the feature data includes the ESF radius of the target image, and the acquisition unit includes a first processing module, a first determination module, a second processing module, a fitting module and a second determination module. The first processing module is used to perform grayscale processing and noise removal processing on the knife edge image to obtain a first processed image; the first determination module is used to determine the initial blade edge in the first processed image by using an edge detection algorithm, and obtain a straight blade edge by fitting the initial blade edge by the least squares method, and the edge detection algorithm is a Laplace algorithm or a Canny edge detector algorithm; the second processing module is used to perform sub-pixel edge positioning on the straight blade edge by using a difference algorithm, and project along the blade edge direction of the straight blade edge to obtain a discrete blade edge brightness distribution curve; the fitting module is used to perform equal-interval sampling on the blade edge brightness distribution curve to obtain sampling point data, and use a fitting function to fit the sampling point data to obtain an edge extension function curve, and the fitting function is a cubic spline interpolation function or a Fermi function; the second determination module is used to determine the ESF radius of the target image according to the edge extension function curve.

[0116] In some examples, the second determination module includes a first determination submodule, a second determination submodule and a first acquisition submodule, wherein the first determination submodule is used to calculate the value of the edge spread function curve according to the maximum value m of the edge spread function curve, the minimum value n of the edge spread function curve, and the first threshold value. And the second threshold calculation formula , respectively determine the first threshold and the second threshold, wherein, is the first threshold, is the second threshold, 0≤a≤1; the second determination submodule is used to determine the point in the edge spread function curve that is closest to the first threshold as the first data point, and determine the point in the edge spread function curve that is closest to the second threshold as the second data point; the first acquisition submodule is used to acquire the distance calculation formula , and calculate the absolute distance between the first data point and the second data point according to the distance calculation formula to obtain the ESF radius, is the ESF radius, is the first data point, is the second data point.

[0117] In some examples, the feature data includes the PSF radius of the target image, and the acquisition unit includes a conversion module, a third processing module, a wavelet transform module, a fourth processing module, a third determination module and a fourth determination module. The conversion module is used to convert the knife-edge image into a grayscale image; the third processing module is used to use the Laplace operator to perform second-order differential processing on the grayscale image to calculate the autocorrelation matrix of the grayscale image to obtain the PSF image, and extract the center data from the autocorrelation matrix, wherein the center data includes a plurality of center sub-data, and the center data is the center row data or the center column data; the wavelet transform module is used to use the wavelet transform formula , perform wavelet transform on each of the central sub-data to obtain the corresponding transformed data, where, is the transformed data after c has been subjected to the wavelet transform, is the wavelet basis used in the wavelet transform, W is the wavelet transform, and c is the central sub-data; the fourth processing module is used to perform threshold processing on all the transformed data, and all the transformed data greater than 0 and the transformed data with an absolute value less than a preset value are determined as 0 to obtain the processed data; the third determining module is used to determine the position of the kth minimum point and the position of the k+1th minimum point in the processed data as the first position point and the second position point respectively; the fourth determining module is used to calculate the radius according to the formula and the position difference between the first position point and the second position point, to determine the PSF radius, wherein, is the PSF radius, is the position of the first position point, is the position of the second position point.

[0118] In some examples, the feature data includes the maximum radius of the target image, the acquisition unit includes a fifth determination module, a calculation module and a sixth determination module, the fifth determination module is used to calculate the third threshold value according to the maximum brightness value of the knife edge image and the minimum brightness value of the knife edge image. and the fourth threshold budget formula , respectively determine the third threshold value and the fourth threshold value, is the third threshold, is the fourth threshold, is the maximum brightness of the knife-edge image, is the minimum brightness value of the knife-edge image, 0≤b≤1; the calculation module is used to calculate the absolute value of the difference between each pixel point in the knife-edge image and the third threshold to obtain the corresponding first absolute value, and calculate the absolute value of the difference between each pixel point in the knife-edge image and the fourth threshold to obtain the corresponding second absolute value; the sixth determination module is used to determine the maximum radius of the target image according to all the first absolute values ​​and all the second absolute values.

[0119] In some examples, the sixth determination module includes a third determination submodule, a fourth determination submodule, a calculation submodule and a fifth determination submodule, the third determination submodule is used to arrange all the first absolute values ​​from small to large, and determine the pixel points corresponding to the first absolute values ​​preset as the first group of feature points; the fourth determination submodule is used to arrange all the second absolute values ​​from small to large, and determine the pixel points corresponding to the second absolute values ​​preset as the second group of feature points; the calculation submodule is used to calculate the distance between any two feature points in the first group of feature points and the second group of feature points to obtain multiple calculated distances; the fifth determination submodule is used to calculate the distance between any two feature points in the first group of feature points and the second group of feature points according to the maximum radius calculation formula , determine the maximum radius of the target image, where, is the maximum radius, is the coordinate of the i-th point in the first set of feature points, is the coordinate of the jth point in the second set of feature points.

[0120] In some examples, the second processing unit includes a first acquisition module, a seventh determination module, a second acquisition module and an eighth determination module, wherein the first acquisition module is used to acquire the relationship model , the relationship model is a relationship model between the feature data of the target image and the defocus position of the target image, wherein, is the ESF radius, is the PSF radius, is the maximum radius, is the defocus position of the target image, a is the first coefficient, b is the second coefficient, c is the third coefficient, and d is the fourth coefficient, wherein d is not zero, and at least one of a, b, and c is not zero; the seventh determination module is used to determine the defocus position of the target image according to the relationship model; the second acquisition module is used to obtain the defocus amount calculation formula ,in, is the target image, is the target defocus direction, is the defocus position of the target image, is the defocus amount of the target image; and the eighth determination module is used to determine the defocus amount of the target image according to the defocus amount calculation formula.

[0121] The image autofocus device includes a processor and a memory, and the acquisition unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0122] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be provided, and the problem of low efficiency and large amount of calculation in calculating the focus position of an image in the prior art can be solved by adjusting kernel parameters.

[0123] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0124] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the automatic focusing method of the image.

[0125] An embodiment of the present invention provides a processor, which is used to run a program, wherein the automatic focusing method of the image is executed when the program is run.

[0126] The embodiment of the present invention provides a device, the device includes a processor, a memory and a program stored in the memory and executable on the processor, and the processor implements the steps of the automatic focusing method of the image when executing the program. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0127] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program of initializing the steps of an auto-focus method having at least an image.

[0128] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0135] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0138] 1) The automatic focusing method of the above-mentioned image of the present application firstly cuts out the blade image from the target image, determines the characteristic data of the target image according to the blade image, and the characteristic data includes at least one of the following: the ESF radius, PSF radius and maximum radius of the target image, and the blade image is an image including a clear blade edge; then, multiple sets of training data are used to obtain an overfocus and underfocus classification neural network based on the FocusNet network training; then, the target image is input into the overfocus and underfocus classification neural network to obtain the target defocus direction of the target image; finally, the defocus amount of the target image is determined according to the characteristic data of the target image and the target defocus direction, and the target image is automatically focused according to the defocus amount. This method uses three characteristic indicators that are linearly related to the defocus amount, and infers the defocus amount through a linear model or a high-order model. Without changing the existing configuration, a single picture is used to quickly predict the quasi-focus position of the chip, reduce time cost, and solve the problem of low efficiency and large amount of calculation in calculating the focus position of the image in the prior art.

[0139] 2) The automatic focusing device for the above-mentioned image of the present application includes an acquisition unit, a training unit, a first processing unit and a second processing unit. The acquisition unit is used to cut out a knife edge image from a target image, and determine the characteristic data of the target image according to the knife edge image. The characteristic data includes at least one of the following: the ESF radius, PSF radius and maximum radius of the target image, and the knife edge image is an image including a clear blade edge; the training unit is used to obtain an overfocus and underfocus classification neural network based on FocusNet network training using multiple sets of training data; the first processing unit is used to input the target image into the overfocus and underfocus classification neural network to obtain the target defocus direction of the target image; the second processing unit is used to determine the defocus amount of the target image according to the characteristic data of the target image and the target defocus direction, and automatically focus the target image according to the defocus amount. The device uses three characteristic indicators that are linearly related to the defocus amount, and inverts the defocus amount through a linear model or a high-order model. Without changing the existing configuration, a single picture is used to quickly predict the quasi-focus position of the chip, reduce time cost, and solve the problem of low efficiency and large amount of calculation in calculating the focus position of the image in the prior art.

[0140] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An automatic focusing method for an image, characterized in that: include: Acquire a target image, and cut out a knife edge image from the target image, determine feature data of the target image according to the knife edge image, the feature data including at least one of the following: an ESF radius, a PSF radius, and an extreme radius of the target image, the knife edge image is an image including a clear knife edge, and the knife edge is a straight line boundary between two adjacent areas whose brightness difference is greater than a preset difference; A neural network for over-focus and under-focus classification is obtained based on FocusNet network training using multiple sets of training data, wherein each set of training data includes: historical images and historical defocus directions corresponding to the historical images acquired in a historical time period; Inputting the target image into the overfocus or underfocus classification neural network to obtain a target defocus direction of the target image; Determining a defocus amount of the target image according to the feature data of the target image and the target defocus direction, and automatically focusing the target image according to the defocus amount; The feature data includes a PSF radius of the target image, and determining the feature data of the target image according to the knife-edge image includes: Converting the knife edge image into a grayscale image; Performing second-order differential processing on the grayscale image using a Laplace operator to calculate an autocorrelation matrix of the grayscale image to obtain a PSF image, and extracting center data from the autocorrelation matrix, wherein the center data includes a plurality of center sub-data, and the center data is center row data or center column data; Using wavelet transform formula , perform wavelet transform on each of the central sub-data to obtain the corresponding transformed data, where, is the transformed data after c has been subjected to the wavelet transform, is the wavelet basis used in the wavelet transform, W is the wavelet transform, and c is the central sub-data; Performing threshold processing on all the transformed data, determining all the transformed data greater than 0 and the transformed data with an absolute value less than a preset value as 0, to obtain processed data; Determine the position of the kth minimum value point and the position of the k+1th minimum value point in the processed data as the first position point and the second position point respectively; According to the radius calculation formula and the position difference between the first position point and the second position point, to determine the PSF radius, wherein, is the PSF radius, is the position of the first position point, is the position of the second position point.

2. The method according to claim 1, characterized in that The feature data includes an ESF radius of the target image, and determining the feature data of the target image according to the knife edge image includes: Performing grayscale processing and noise removal processing on the knife edge image to obtain a first processed image; An initial edge in the first processed image is determined by using an edge detection algorithm, and a linear edge is obtained by fitting the initial edge by a least square method, wherein the edge detection algorithm is a Laplace algorithm or a Canny edge detector algorithm; Using a difference algorithm to perform sub-pixel edge positioning on the straight edge, and projecting along the edge direction of the straight edge to obtain a discrete edge brightness distribution curve; The blade edge brightness distribution curve is sampled at equal intervals to obtain sampling point data, and the sampling point data is fitted with a fitting function to obtain an edge extension function curve, wherein the fitting function is a cubic spline interpolation function or a Fermi function; The ESF radius of the target image is determined according to the edge spread function curve.

3. The method according to claim 2, characterized in that Determining the ESF radius of the target image according to the edge spread function curve includes: According to the maximum value m of the edge spread function curve, the minimum value n of the edge spread function curve, and the first threshold calculation formula And the second threshold calculation formula , respectively determine the first threshold and the second threshold, wherein, is the first threshold, is the second threshold, 0≤a≤1; Determine a point in the edge spread function curve that is closest to the first threshold as a first data point, and determine a point in the edge spread function curve that is closest to the second threshold as a second data point; Get the distance calculation formula , and calculate the absolute distance between the first data point and the second data point according to the distance calculation formula to obtain the ESF radius, is the ESF radius, is the first data point, is the second data point.

4. The method according to claim 1, characterized in that The feature data includes the maximum radius of the target image, and determining the feature data of the target image according to the knife edge image includes: According to the maximum brightness value of the knife-edge image and the minimum brightness value of the knife-edge image, the third threshold calculation formula is: and the fourth threshold budget formula , respectively determine the third threshold value and the fourth threshold value, is the third threshold, is the fourth threshold, is the maximum brightness of the knife-edge image, is the minimum brightness value of the knife edge image, 0≤b≤1; Calculating the absolute value of the difference between each pixel point in the knife-edge image and the third threshold to obtain a corresponding first absolute value, and calculating the absolute value of the difference between each pixel point in the knife-edge image and the fourth threshold to obtain a corresponding second absolute value; The maximum radius of the target image is determined according to all the first absolute values ​​and all the second absolute values.

5. The method according to claim 4, characterized in that Determining the maximum radius of the target image according to all the first absolute values ​​and all the second absolute values ​​includes: Arrange all the first absolute values ​​from small to large, and determine the pixel points corresponding to the first absolute values ​​preset before as a first group of feature points; Arrange all the second absolute values ​​from small to large, and determine the pixel points corresponding to the first preset second absolute values ​​as the second group of feature points; Calculating the distance between any two feature points in the first group of feature points and the second group of feature points to obtain multiple calculated distances; According to the maximum radius calculation formula , determine the maximum radius of the target image, where, is the maximum radius, is the coordinate of the i-th point in the first set of feature points, is the coordinate of the jth point in the second set of feature points.

6. The method according to claim 1, characterized in that Determining the defocus amount of the target image according to the feature data of the target image and the target defocus direction includes: Get the relationship model , the relationship model is a relationship model between the feature data of the target image and the defocus position of the target image, wherein, is the ESF radius, is the PSF radius, is the maximum radius, is the defocus position of the target image, a is the first coefficient, b is the second coefficient, c is the third coefficient, and d is the fourth coefficient, wherein d is not zero, and at least one of a, b, and c is not zero; Determining a defocus position of the target image according to the relationship model; Get the defocus calculation formula ,in, is the target image, is the target defocus direction, is the defocus position of the target image, is the defocus amount of the target image; The defocus amount of the target image is determined according to the defocus amount calculation formula.

7. An automatic focusing device for an image, characterized in that: include: an acquisition unit, configured to acquire a target image, and cut out a knife-edge image from the target image, and determine feature data of the target image according to the knife-edge image, wherein the feature data includes at least one of the following: an ESF radius, a PSF radius, and an extreme radius of the target image, wherein the knife-edge image is an image including a clear knife edge, and the knife edge is a straight line boundary between two adjacent areas whose brightness difference is greater than a preset difference; A training unit, for obtaining an over-focus and under-focus classification neural network based on FocusNet network training using multiple sets of training data, wherein each set of training data in the multiple sets of training data includes: historical images and historical defocus directions corresponding to the historical images acquired in a historical time period; A first processing unit is used to input the target image into the overfocus or underfocus classification neural network to obtain a target defocus direction of the target image; a second processing unit, configured to determine a defocus amount of the target image according to the feature data of the target image and the target defocus direction, and automatically focus the target image according to the defocus amount; The feature data includes the PSF radius of the target image. The acquisition unit includes a conversion module, a third processing module, a wavelet transform module, a fourth processing module, a third determination module and a fourth determination module. The conversion module is used to convert the knife edge image into a grayscale image; the third processing module is used to use the Laplace operator to perform second-order differential processing on the grayscale image to calculate the autocorrelation matrix of the grayscale image to obtain the PSF image, and extract the center data from the autocorrelation matrix. The center data includes a plurality of center sub-data, and the center data is the center row data or the center column data; the wavelet transform module is used to use the wavelet transform formula , perform wavelet transform on each of the above central sub-data to obtain the corresponding transformed data, where, is the transformed data after c has been transformed by the above wavelet transform, is the wavelet basis used in the above wavelet transform, W is the above wavelet transform, and c is the above central sub-data; the fourth processing module is used to perform threshold processing on all the above transformed data, and all the above transformed data greater than 0 and the above transformed data with absolute values ​​less than the preset value are determined as 0 to obtain processed data; the third determination module is used to determine the position of the kth minimum point and the position of the k+1th minimum point in the above processed data as the first position point and the second position point respectively; the fourth determination module is used to calculate the radius according to the formula and the position difference between the first position point and the second position point, to determine the PSF radius, wherein, is the PSF radius mentioned above, is the position of the first position point mentioned above, is the position of the second position point mentioned above.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the automatic focusing method of an image according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the autofocus method of the image described in any one of claims 1 to 6.

Images