An auto-focusing device and method based on digital image processing

Abstract

The application belongs to the technical field of machine vision and automatic focusing, and particularly relates to an automatic focusing device and method based on digital image processing, which comprises the following steps: receiving a pixel stream and generating corresponding pixel coordinates, processing the pixel stream in a RAW domain to construct pseudo-grayscale, and obtaining denoised pseudo-grayscale by denoising; selecting a focusing window ROI with the maximum texture intensity; generating an ROI gate signal based on the currently selected focusing window ROI; respectively accumulating three paths of frame-level scores under the ROI gate signal; processing the three paths of frame-level scores to obtain a final sharpness score; and updating the motion direction and the step pulse frequency of a stepping motor in cycles of frames according to the inter-frame variation trend of the final sharpness score. The application realizes automatic focusing under the condition of continuous motion of a stepping motor.

Description

Technical Field

[0001] This application belongs to the field of machine vision and autofocus technology, specifically an autofocus device and method based on digital image processing. Background Technology

[0002] Existing autofocus systems mostly employ a discrete control method involving motor movement, stopping, image acquisition, sharpness calculation, and re-movement. This method suffers from low focusing efficiency, significant mechanical shock, and susceptibility to jitter. Furthermore, some solutions rely on ISP de-mosaicing and RGB / grayscale reconstruction before sharpness evaluation, resulting in long computational chains and high hardware resource consumption. In the presence of readout noise or quantization noise, sharpness scores are prone to jitter or false peaks, leading to unstable focusing decisions. For applications requiring implementation on the FPGA side and real-time completion of "continuous acquisition, continuous scoring, and continuous speed / direction adjustment" during continuous motor movement, the above solutions struggle to balance real-time performance, stability, and hardware feasibility. Therefore, it is necessary to provide an autofocus solution with a clear implementation path, controllable resources, and suitability for real-time FPGA implementation. Summary of the Invention

[0003] This application provides an autofocus device and method based on digital image processing, which solves the problems of low focusing efficiency, large start-stop shock, and unstable control caused by scoring jitter.

[0004] An autofocus method based on digital image processing, according to an embodiment of the first aspect of this application, includes: receiving a pixel stream and generating corresponding pixel coordinates to form a single continuous pixel stream data path;

[0005] In the RAW domain, a pseudo grayscale is constructed by processing the pixel stream, and the pseudo grayscale is then denoised to obtain a denoised pseudo grayscale.

[0006] Five focus windows are fixed, and five paths corresponding to the ROI of each focus window are generated based on pseudo grayscale and pixel coordinates. Gating signal, The value ranges from 1 to 5, and is used in accordance with the above. The gating signal calculates the texture intensity of the five ROIs in the focus window. When the texture intensity of the currently selected ROI is lower than a preset threshold, it switches to the ROI with the highest texture intensity.

[0007] Generate an ROI gating signal based on the currently selected ROI focus window;

[0008] Within the currently selected ROI focus window, three pixel-wise feature values ​​are simultaneously calculated for the denoising pseudo-grayscale of the pixel stream, and then accumulated into three frame-level scores under the ROI gating signal.

[0009] Logarithmically compress and normalize the three frame-level scores to obtain three 8-bit score values. Then, fuse the three 8-bit score values ​​with fixed weights and perform first-order IIR smoothing on the fused result to obtain the final sharpness score.

[0010] Based on the inter-frame variation trend of the final sharpness score, the movement direction and step pulse frequency of the stepper motor are updated on a frame-by-frame basis.

[0011] Furthermore, the pseudo grayscale satisfies the following: when the pixel coordinates are at the position of a green sampling point, the pseudo grayscale is equal to the pixel coordinates; when the pixel coordinates are at the position of a red or blue sampling point, the pseudo grayscale is the average of the four neighboring green pixels of the pixel coordinates.

[0012] Furthermore, statistical texture intensity includes:

[0013] Denoising pseudo-grayscale values ​​are convolved with the Sobel operator to obtain the horizontal and vertical gradients;

[0014] The gradient magnitude is obtained by summing the absolute values ​​of the horizontal and vertical gradients.

[0015] A texture indication function is constructed based on the gradient magnitude;

[0016] For each focus window ROI, a texture indication function is used, in Parallel counting of strong edge pixels under gated signals;

[0017] The texture intensity is determined by the ratio of the number of pixels with strong edges to a fixed constant or by the number of pixels with strong edges.

[0018] Furthermore, the three-path frame-level scores include:

[0019] Score A: The Sobel operator is applied to the denoised pseudo-grayscale to obtain the horizontal and vertical gradients, and the absolute values ​​of the horizontal and vertical gradients are accumulated within the pixel clock of the ROI gate signal to obtain the frame-level gradient energy score.

[0020] Score B: Calculate the same-color difference energy of the pixel at the green sampling point and accumulate it within the pixel clock when the ROI gate signal is valid to obtain the frame-level score;

[0021] Score C: Calculate the horizontal and vertical Laplacian responses of the denoised pseudo-grayscale, and accumulate the horizontal and vertical Laplacian responses within the pixel clock of the ROI-gated signal to obtain the frame-level improved Laplacian energy and score.

[0022] Furthermore, the three frame-level scores are logarithmically compressed and normalized, including: obtaining the exponent e by detecting the first 1, taking the 6-bit mantissa immediately following the most significant bit as the index, and outputting an 8-bit score value after logarithm compression and normalization by querying a 64×8-bit lookup table to obtain the compensation value.

[0023] Furthermore, a first-order IIR smoothing is performed on the fusion result to obtain the final sharpness score, including:

[0024] The smoothed output is calculated as follows: ,in for Sub-smooth output. for Sub-smooth output. , It is an 8-bit fused value. b=3 is equivalent to α=1 / 8, and a high-pass residual is constructed within the current focus window ROI to calculate the noise intensity index. When the noise intensity index is greater than the set threshold, b is adjusted to 4.

[0025] Furthermore, the movement direction and step pulse frequency of the stepper motor are updated on a frame-by-frame basis, including: using three states—coarse search, fine search, and lock—to update and switch frame by frame. The coarse search quickly sweeps across the depth of field at a high step frequency and continuously records the maximum score position pos_max; when the score trend indicates that the peak has been exceeded, the fine search is entered, and a low-frequency micro-scan is performed near the maximum score position pos_max, while continuously updating the maximum score position pos_max; when the score change enters the stable plateau range, the maximum score position pos_max is returned to, and the step pulse output is stopped to complete the lock.

[0026] An autofocus device based on digital image processing, according to a second aspect of this application, includes:

[0027] FPGA processor, the FPGA processor being configured to:

[0028] The image acquisition module is used to receive pixel streams;

[0029] The data unpacking and line / field reconstruction module is used to generate corresponding pixel coordinates from the pixel stream to form a single continuous pixel stream data path.

[0030] The pseudo-grayscale construction module is used to process pixel streams in the RAW domain to construct pseudo-grayscale values;

[0031] A 3x3 denoising filter module is used to denoise the pseudo grayscale to obtain a denoised pseudo grayscale.

[0032] The focus window selection and ROI gating module is used to fix five focus windows and generate five paths corresponding to the ROI of each focus window based on pseudo grayscale and pixel coordinates. Gating signal, The value ranges from 1 to 5, and is used in accordance with the above. The gating signal calculates the texture intensity of the five ROIs in the focus window. When the texture intensity of the currently selected ROI is lower than a preset threshold, it switches to the ROI with the highest texture intensity. An ROI gating signal is generated based on the currently selected ROI.

[0033] The sharpness evaluation engine is used to calculate three pixel-wise feature values ​​for the pixel stream in the current selected ROI (Region of Interest) for denoising pseudo-grayscale synchronization, and accumulate them into three frame-level scores under the ROI gate signal.

[0034] The logarithmic compression and normalization module performs logarithmic compression and normalization on the three frame-level scores to obtain three 8-bit score values.

[0035] The fixed-weight fusion module is used to fuse the 8-bit scores from the three channels according to fixed weights.

[0036] The first-order IIR smoothing module is used to perform first-order IIR smoothing on the fusion result to obtain the final sharpness score;

[0037] The focus and motor control module is used to update the movement direction and step pulse frequency of the stepper motor on a frame-by-frame basis, based on the inter-frame variation trend of the final sharpness score.

[0038] Furthermore, the focusing and motor control module includes:

[0039] The system includes a stepping pulse generator, a position counter, and a focus control state machine. The stepping pulse generator continuously outputs signals based on the speed register and the direction register to drive the stepping motor. The focus control state machine reads the smooth output at each frame boundary and updates the speed register and the direction register.

[0040] Compared with the prior art, the advantages of this application are as follows:

[0041] This application enables automatic focusing under continuous stepper motor movement conditions, in order to overcome the problems of low efficiency, large start-stop shock, and unstable control caused by score jitter in the existing discrete "walk-stop-shoot-calculate-walk again" focusing method, thereby improving focusing speed and peak lock reliability.

[0042] The focusing method, which uses continuous motion and updates control values ​​frame by frame, avoids the mechanical shock and efficiency loss of discrete focusing, thereby improving focusing speed and control smoothness.

[0043] By constructing and denoising pseudo-grayscale values ​​in the RAW domain, the jitter effect of noise on the sharpness evaluation score is suppressed, thereby reducing false peaks and misjudgments.

[0044] This application can balance trend stability, RAW determinism, and peak sharpness, thereby improving the reliability of peak positioning;

[0045] Image preprocessing, sharpness evaluation, score normalization fusion, smoothing filtering, and motor focus control are completed through a single-path pixel pipeline. Only line buffers and frame-level accumulator registers are used to avoid the storage and bandwidth pressure caused by full-frame buffering, thereby meeting the real-time processing requirements under high throughput conditions such as 1920×1080@60fps. Attached Figure Description

[0046] Figure 1 A general structural block diagram of an autofocus device based on digital image processing provided in this application embodiment;

[0047] Figure 2 This is a block diagram showing the connection relationship between the autofocus processing link and functional modules inside the FPGA processor. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] Terminology, Symbols and Implementation Conventions:

[0050] To facilitate understanding of the implementation of this application and avoid misinterpretation of hardware semantics, this embodiment makes the following stipulations regarding the terminology, symbols, and implementation methods, which are used to provide a consistent description of the inputs, outputs, and computational objects of subsequent modules.

[0051] (1) Pixel flow and coordinates: RGGB RAW12 single-channel pixel stream used to represent the output of an image sensor in pixel coordinates Pixel value at; pixel coordinates Generated by an FPGA (Field Programmable Gate Array) in scanning order under line / field synchronization constraints, it provides a unified index for ROI (Region of Interest) boundary determination, sliding window construction, and frame-level accumulation gating, so that subsequent processing links can all work based on the same coordinate system, thereby ensuring consistency of the computational objects.

[0052] (2) Pseudo-grayscale and denoised pseudo-grayscale: Y raw (x,y) is used to represent the pseudo-grayscale of a single channel constructed in the RAW domain; Y lp (x, y) is used to represent the relationship between Y.raw (x,y) is the denoised pseudo-grayscale value after performing a fixed 3×3 kernel denoising filter. raw (x,y) and Y lp (x,y) is used to stably extract high-frequency detail information and calculate the sharpness score without performing demosaicing and RGB (red, green, and blue primary colors) reconstruction, thereby reducing link complexity and improving hardware determinism.

[0053] (3) Pixel-by-pixel feature value and frame-level score: The pixel-by-pixel feature value is used to represent the feature quantity calculated on the current arriving pixel in the pixel pipeline; the frame-level score is used to represent the statistical quantity obtained by gating and accumulating the pixel-by-pixel feature values ​​of the effective pixels within the currently selected focus window ROI and latching the output at the end of the frame. This convention is used to clarify the statistical path of "point-by-point calculation - gating accumulation - frame-end latching" in order to distinguish between pixel-by-pixel operations and frame-level statistics, thereby avoiding the misunderstanding that the entire frame needs to be buffered to obtain the score.

[0054] (4) Parallel hardware semantics: In this embodiment, "parallel" refers to the synchronous calculation of multiple operators on the same single pixel stream. Parallelism in focus window selection and ROI gating is only used for texture statistics of multiple focus windows; sharpness scores A / B / C and fusion / IIR are only based on the currently selected focus window ROI region.

[0055] (5) Threshold selection: The noise intensity index Nr comparison threshold, texture insufficiency judgment threshold, and texture indicator function threshold are preferred engineering constants under the conditions of RAW12, fixed 3×3 denoising and 200×150 focus window ROI; without changing the core process of this application, they can also be adjusted within a small range by calibration according to the image sensor noise level, gain setting or application scenario.

[0056] (6) Arithmetic operations: Score value of the current frame Change in score Signed fixed-point representation (two's complement) is used, and both difference and right shift are implemented according to the arithmetic terminology to support the negative value criterion.

[0057] See Figure 1 This embodiment provides an autofocus device based on digital image processing, including a lens assembly 100, an image sensor 200, a stepper motor 300 and driver 310, a homing reference sensor 320, and an FPGA processor 400.

[0058] The lens assembly 100 has a focusing mechanism that allows for relative displacement along the optical axis. Preferably, a stepper motor combined with a lead screw / guide rail slide is used to convert rotational motion into axial displacement of the lens, thereby achieving focusing.

[0059] Image sensor 200 outputs RAW Bayer image data, preferably in RGGB (standard Bayer color filter array) format, with a resolution of 1920×1080, a frame rate of 60fps, and an output bit width of RAW12 (single-plane, 12-bit per pixel, high-density packed raw Bayer format). The image sensor 200 and FPGA processor 400 can communicate via MIPI CSI-2, LVDS, SLVS, or SLVS-EC interfaces. FPGA processor 400 can initialize and configure the image sensor 200's exposure, gain, frame rate, and black level clamp registers via I2C / SPI (Integrated Circuit Interconnect Bus / Serial Peripheral Interface) to ensure stable and consistent RAW12 output, thereby reducing the risk of subsequent sharpness scoring jitter.

[0060] The focusing actuator is preferably a two-phase or multi-phase stepper motor 300, and the driver 310 preferably supports microstepping drive (e.g., 1 / 16 microstepping). The FPGA processor 400 outputs STEP / DIR / EN signals to control the start / stop, continuous rotation, commutation, and speed adjustment of the stepper motor 300. The device is equipped with a zero-return reference sensor 320 (photoelectric / Hall / limit switch are all acceptable). Upon power-up, the FPGA processor 400 drives the stepper motor 300 to perform a zero-return action, and clears the position count register to zero when the reference is triggered, defining pos=0 (position equals zero). Subsequently, the open-loop counting of the focusing position is achieved by accumulating STEP pulses (single-wire digital signals triggered by the rising edge). It should be noted that: without the configuration of position feedback devices such as encoders, the position pos is obtained by pulse accumulation, which is an open-loop position counting; the feedback closed loop of this application is preferably embodied as a focusing effect feedback closed loop based on sharpness score, that is, the direction of the stepper motor 300 and the pulse frequency are updated frame by frame according to the score trend, so as to achieve focus convergence and locking during continuous movement.

[0061] See Figure 2 The FPGA processor 400 internally includes at least: an image acquisition module 401, a data unpacking and line / field reconstruction module 402, a pseudo grayscale construction module 410, a 3x3 noise reduction and filtering module 420, a focus window selection and ROI gating module 430, a sharpness evaluation engine 450, a logarithmic compression and normalization module 460, a fixed weight fusion module 470, a first-order IIR smoothing module 480, and a focus and motor control module 490.

[0062] The image acquisition module 401 receives the pixel stream and completes image data packet reception. The data unpacking and line / field reconstruction module 402 generates corresponding pixel coordinates from the pixel stream to form a single continuous pixel stream data path. It completes bitstream unpacking and line / field synchronous reconstruction, and outputs the single pixel stream, pixel valid handshake signal pix_valid, and pixel coordinates (x, y) point by point under the pixel clock beat. Subsequent circuits process the single pixel stream in a pipelined order to avoid whole-frame buffering.

[0063] See Figure 2 The FPGA processor 400 contains a pseudo-grayscale construction module 410, which constructs a single-channel pseudo-grayscale in the RAW domain. This method is used to eliminate spurious responses caused by Bayer lattice structures to high-frequency operators (gradients and Laplacian types). The construction does not perform demosaicing or RGB reconstruction, is computationally deterministic, and has low resource overhead. Its rules are as follows:

[0064] If pixel coordinates (x, y) are the green sampling point, then That is, pseudo grayscale is equal to pixel value;

[0065] If pixel coordinates (x, y) are a red or blue sampling point, then the average green value of the four neighbors of pixel coordinates (x, y) is taken:

[0066] ,

[0067] The four neighboring green areas can be determined by the parity of the pixel coordinates and provided by the row buffer / shift register. pixel coordinates pixel values, pixel coordinates pixel values, pixel coordinates pixel values, pixel coordinates The pixel value.

[0068] See Figure 2 The FPGA processor 400 internally includes a 3×3 noise reduction filter module 420 to filter pseudo-grayscale Y. raw Perform a fixed 3×3 filter on (x,y) and output a denoised pseudo-grayscale result. The filter kernel is fixed as a Gaussian approximation kernel:

[0069] ,

[0070] The filtering multiplication by 2 and 4 is implemented using bit shifting, while division by 16 is implemented by right shifting by 4 bits. The 3×3 sliding window is constructed using two row buffers and a shift register, and a pipelined approach is used to output a filtering result per pixel clock cycle, i.e., denoising pseudo-grayscale. .

[0071] See Figure 2 The FPGA processor 400 is internally equipped with a focus window selection and ROI gating module 430. The focus window selection and ROI gating module 430 selects the currently used focus window ROI according to the texture intensity of different focus window ROIs and outputs the ROI gating signal of the focus window ROI.

[0072] In this embodiment, five focus windows are fixed, namely the center... Top left Top right Bottom left Bottom right Each ROI is 200×150 in size, and its top-left corner coordinates are as follows: center Top left Top right Bottom left Bottom right .

[0073] The focus window selection and ROI gating module 430 compares the pixel coordinates (x, y) with the boundary coordinates of each focus window's ROI to generate... Gating signal, Values ​​range from 1 to 5, and a valid signal is denoted as... Gating signal, Gating signals include Gating signal. The gating signal is used to gate the subsequent texture intensity statistics module 431, and the value of i is 1-5: when When the gating signal is valid, the texture indicator function b(x,y) is accumulated into the texture accumulation register corresponding to the ROI of the focus window. ;when When the gating signal is invalid, the corresponding texture accumulation register... This remains unchanged. Therefore, during the scanning of one frame using a single pixel stream, texture statistics for the five focus window ROIs can be obtained simultaneously, and texture intensity can be latched at the end of the frame. There is no need to crop multiple image streams or perform full-frame buffering. The gating signal is the gating signal for the currently used focus window ROI, which is equivalent to the gating signal for the currently used focus window corresponding to... Gating signal. Focus window ROI selection and ROI gating module 430 output. The gating signal is sent to the sharpness evaluation engine module 450 to gating the accumulation and latching of subsequent sharpness evaluation scores A / B / C, so as to ensure that subsequent normalization, fusion, first-order IIR smoothing and motor closed-loop control only use the statistical results of the current focus window ROI as input.

[0074] Texture intensity was statistically analyzed for five focus window ROIs. Texture intensity was defined as the percentage of pixels within a focus window ROI whose gradient response exceeded a threshold, specifically including:

[0075] Denoising pseudo grayscale Obtain the horizontal gradient by performing Sobel convolution. and vertical gradient :

[0076] ,

[0077] ,

[0078] Horizontal gradient and vertical gradient The gradient magnitude is obtained by summing the absolute values ​​of the gradients: ;

[0079] Define texture indicator function (threshold) With a fixed constant of 8, the threshold is selected to be higher than the noise gradient bottom of the denoised flat region and lower than the engineering boundary of the typical edge gradient response, in order to suppress noise-induced false edges and maintain the stability of texture statistical caliber.

[0080] ,

[0081] This is a texture indicator function.

[0082] For each focus window ROI, in Parallel counting of strong edge pixels under gated signal :

[0083] ,

[0084] Indicates the first One focus window.

[0085] Texture intensity is calculated as the percentage of pixels with strong edges ( =200x150 is a fixed constant):

[0086] ,

[0087] In this paper, texture intensity can be represented as the number of strong edge pixels in count form. (k) can also be expressed as the percentage of strong edge pixels N in percentage form. (k), the two are equivalent and convertible. Furthermore, in this embodiment... =200x150 is a fixed constant. To facilitate hardware implementation on the FPGA processor 400, the texture intensity used for calculation in this paper will be based on the number of pixels at strong edges. express.

[0088] This embodiment defaults to setting the center focus window ROI1 as the initial selected focus window. To avoid continuous motion loop jitter caused by frequent ROI switching, this embodiment adopts a simplified switching strategy: The texture intensity of the current focus window ROI is set to... ,when When three consecutive frames are achieved, the focus window switching module 432 switches to the focus window ROI with the highest texture intensity and holds it for six frames without switching again. This avoids control jitter caused by frequent focus window switching in continuous motion loops. A strong edge pixel count threshold of 2000 is set, which corresponds to approximately 6–8% strong edge coverage within the focus window ROI. Below this coverage, sharpness evaluation is more easily dominated by noise, so insufficient texture is determined and window switching is triggered.

[0089] See Figure 2 The sharpness evaluation engine 450 includes a score A (gradient energy) operator, a score B (chromatic difference) operator, a score C (improved Laplacian energy sum) operator, and a frame-level accumulation register. The three scoring operators operate synchronously and in parallel within the same clock domain: calculating pixel-by-pixel feature values ​​for a single pixel stream input and accumulating them to the corresponding frame-level accumulation registers under ROI gating signals. , , In the middle; at the end of each frame, the frame-level accumulator register is latched and output to form three-way frame-level energy scores. , S C (k). The three-way frame-level accumulator registers have a fixed width of 32 bits and are unsigned to avoid overflow when accumulating within a 200×150 focus window ROI.

[0090] Score A: Sobel gradient energy (an image sharpness evaluation metric based on the Sobel operator).

[0091] Denoising pseudo grayscale Obtain the horizontal gradient by performing Sobel operator convolution. and vertical gradient And an approximate amplitude is obtained using hardware:

[0092] ,

[0093] Within the pixel clock when the ROI gate signal is valid, Perform intra-frame accumulation to obtain frame-level gradient energy scores. This score is sensitive to changes in edges and textures, making it suitable as a trend criterion in continuous motion focusing. The above process can be expressed as follows:

[0094] ;

[0095] Score B: Green color difference energy:

[0096] To avoid the red, green, and blue interleaving distribution of the Bayer array causing adjacent pixels to come from different color channels and thus introducing color differences (rather than pure sharpness variations) into the difference results, this embodiment limits the difference operation to the set of green sampling points. The process is performed internally, meaning only "color difference" is performed. Here, the green channel is no longer understood as a complete interpolated image, but rather the value of the RAW pixel stream at the green sampling location is used directly for calculation: when the pixel position... At that time, the pixel value of the green sampling point is the RAW pixel stream. The value at that position; and when the pixel position These are not included in the calculation of this channel. Taking the RGGB color filter array and the top-left pixel position (0,0) as R as an example, the set of green sampling points can be represented as:

[0097] ,

[0098] At the green sampling point set Above, that is, in At that time, use color difference energy with a step size of 2 (to ensure that it still falls on the green sampling grid point):

[0099] ,

[0100] Within the pixel clock when the ROI-gated signal is valid (within the selected focus window ROI), D(x,y) is accumulated pixel by pixel to obtain the frame-level chromatic difference energy score. :

[0101] ,

[0102] The step size of 2 means that for RGGB grid points, green sampling points appear repeatedly on the checkerboard grid at intervals of one pixel. Therefore, a displacement of ±2 still falls within the same type of green sampling point, thus achieving color difference. This score does not depend on de-mosaicing and RGB links, has a short calculation path and strong determinism, and is suitable for real-time focusing scenarios with continuous stepper motor movement.

[0103] Score C: Improved Laplace SML (Improved Laplace Energy Sum):

[0104] Denoising pseudo-grayscale The improved Laplacian response is calculated by taking the absolute values ​​of the second-order differences in the horizontal and vertical directions respectively, and then summing them. The frame-level improved Laplacian energy and score are obtained by accumulating these values ​​within the pixel clock of the ROI-gated signal. This score provides a sharper peak shape, which can enhance the resolution of peak locking during the fine search phase.

[0105] ,

[0106] ,

[0107] ,

[0108] in, For a horizontal Laplace response, It is a vertical Laplace response.

[0109] The three frame-level scores are logarithmically compressed and normalized, the three 8-bit scores are fused with fixed weights, and the fused result is subjected to first-order IIR smoothing to obtain the final sharpness score.

[0110] See Figure 2 The FPGA processor 400 internally includes a logarithmic compression normalization module 460, a fixed-weight fusion circuit 470, and a first-order IIR smoothing module 480 to perform normalization, fusion, and smoothing on the three frame-level scores latched at the end of the frame. The purpose is to stably compress the large-range accumulated scores into 8 bits for easier comparison and fusion, and to suppress inter-frame jitter to improve continuous control stability.

[0111] In this process, logarithmic compression normalization is implemented using LOD + small LUT. The three frame-level scores, denoted by S (S>0), are used to perform Leading-One Detector (LOD) to obtain the position of the most significant bit, e (i.e., ...). In the hardware implementation of the FPGA processor 400, e can be obtained by counting leading zeros: e = 31 − LZC(S), where LZC represents the counting of leading zeros.

[0112] The 6-bit mantissa immediately following the most significant bit is used as the index m∈[0,63], and the fractional compensation is obtained by looking up the frac_lut compensation table. The frac_lut compensation table is stored in the ROM memory of the FPGA processor and has a size of 64x8 bits. A fixed scaling left shift of 4 bits is used, and 0~255 is saturated and truncated by clipping, outputting an 8-bit normalized value. for:

[0113] ,

[0114] This yields the three 8-bit normalized scores. , , .

[0115] This embodiment uses fixed weights to fuse the three normalized scores to obtain a stable overall sharpness evaluation result with a minimal number of parameters. Preferably, the three weights are selected as 0.4 / 0.3 / 0.3, which slightly dominates the trend stability of the gradient channel while taking into account the sampling domain determinism of the RAW green color difference channel and improving the peak sharpness of the Laplace SML channel, thus achieving complementary advantages. To facilitate FPGA hardware implementation and reduce computational complexity, this embodiment uses a fixed-point form with a denominator of 256, i.e., the numerators of the weights are 102 / 77 / 77, satisfying 102+77+77=256. The fused output is defined as:

[0116] ,

[0117] The ">>8" is used to normalize the denominator 256. It is an 8-bit fused value. Compared to performing floating-point operations directly, the FPGA processor 400 uses a fixed-point form with a denominator of 256, which can be completed only through fixed-point multiplication and shifting, thereby improving real-time performance and reducing logic resource consumption.

[0118] In one embodiment, the first-order IIR smoothing circuit 480 expands the 8-bit fused value S8(k) to a Q8.8 fixed-point format to achieve fractional weight smoothing. The Q8.8 specifies the point format: the high 8 bits are the integer part, and the low 8 bits are the fractional part, with a value equal to the original code / 256. Therefore, scale expansion is performed first. (That is, mapping an 8-bit integer to Q8.8), the smooth output is defined as:

[0119] ,

[0120] The above ">>3" is equivalent to α=1 / 8, which makes the output S smooth. f (k) is equivalent to exponential smoothing of the 8-bit fused value by approximately 8 frames of time constant: at 60fps, this can be used to suppress inter-frame jitter caused by noise / vibration while maintaining the trackability of focus trend changes. Furthermore, α=1 / 8 can be achieved through bit shifting, thereby reducing multiplier resources and the number of parameters. Optionally, a high-pass residual can be constructed within the current focus window ROI, and noise intensity metrics can be statistically analyzed. If the noise intensity index If the value exceeds a certain threshold, the right shift is changed from 3 to 4 (α = 1 / 16) to enhance smoothness. The final smoothed output... The score is used as the input score for the focus control state machine.

[0121] Optionally, the noise intensity index statistics module 440 is used to construct a high-pass residual within the current focus window ROI and to statistically analyze the noise intensity index. This is to enhance the first-order IIR smoothing intensity when noise is high, thereby suppressing fusion score jitter. The preferred high-pass residual is defined as:

[0122] ,

[0123] Within the pixel clock when the ROI-gated signal is valid, the high-pass residual H(x,y) is accumulated intra-frame and latched at the end of the frame to obtain the noise intensity index Nr:

[0124] ,

[0125] This noise intensity index It does not participate in the sharpness evaluation itself, but is only used for the selection of two smoothing shift parameters for first-order IIR smoothing, so as to further suppress the jitter of the final output score when the noise is high. In this embodiment, if the noise intensity index If the value is greater than 8, then the right shift amount will be changed from 3 to 4. To enhance smoothness. The noise threshold corresponds to the engineering boundary where the high-pass residual statistics enter a significantly elevated range; beyond this boundary, inter-frame random fluctuations in the fusion score will significantly affect the ΔS criterion. Therefore, the first-order IIR smoothing is derived from... Switch to To enhance noise resistance stability.

[0126] See Figure 2 The focus and motor control module 490 includes a stepper pulse generator 492, a position counter 493, and a focus control state machine 491. The stepper pulse generator 492 continuously outputs signals to drive the stepper motor based on the speed register and direction register. These signals include STEP, DIR (direction signal), and EN (enable signal). The focus control state machine 491 reads and smoothly outputs signals at each frame boundary. It updates the speed and direction registers to ensure the image pipeline continues to rotate without obstructing the motor, enabling continuous motion focusing through continuous acquisition, scoring, speed adjustment, and direction adjustment. The focus control state machine 491 employs coarse search, fine search, and locking mechanisms.

[0127] (1) Zeroing and Position Counting: After power-on, the focus control state machine 491 controls the stepper motor to move in the zeroing direction, triggering the zeroing reference sensor 320 and clearing the position counter 493, defining pos=0. During subsequent focusing, the position pos is accumulated through STEP pulses and used to construct the position-score sequence. At the same time, maintain the maximum score. Its corresponding position of maximum score :like Then update and update This position, pos, is used for subsequent fine-grained search regression and locking. It is obtained by accumulating pulses and is an open-loop counting position.

[0128] (2) Coarse search: In the coarse search stage, the step frequency is fixed at 1. And maintain continuous motion; calculate at the boundaries of each frame. , This is the score value for the current frame. This is the score from the previous frame. This represents the change in score, and the maximum score is continuously updated. And the position of the maximum score, pos_max.

[0129] (3) Peak crossing criterion: when When two consecutive frames are true, the condition is considered to have passed the peak value, and a finer search is initiated. It should be noted that: The Q8.8 fixed point is used, and -512 corresponds to -2.0 (that is, the 8-bit score drops by about 2 code values ​​after fusion). Two consecutive frames are used to resist noise misjudgment, thereby avoiding premature shifting when the score fluctuates slightly.

[0130] (4) Fine search: In the fine search stage, the step frequency f_low = 5kHz, and continuous micro-scanning is performed around the maximum score position pos_max. The micro-scanning range is... During the micro-scan, the score value S of the current frame is compared according to the same rules. f (k) and the maximum score value Smax, and continue to update the maximum score value Smax and the maximum score position pos_max when a larger score appears, so as to record the true peak position within the micro-scan range as the final locked target.

[0131] (5) Locking Criterion: When |ΔS(k)|<128 is true for 6 consecutive frames, it is determined that the peak plateau has been entered. The stepper motor is controlled to return to the maximum score position pos_max and the output of step pulses is stopped to complete the lock. The value 128 corresponds to 0.5 in Q8.8 (that is, the change of 8-bit fusion value is less than about 0.5 code values), indicating that the score change has entered the plateau stable area. At this time, the focus control state machine controls the stepper motor to return to the maximum score position pos_max at a fine search frequency and stops outputting step pulses to complete the focus lock, thereby reducing the risk of repeated reversal under noise.

[0132] The image processing pipeline operates on a pixel clock, while the stepper pulse generator 492 operates on an independent timer / divider clock. The focus control state machine 491 updates the speed and direction registers at each frame boundary, and the stepper pulse generator 492 reads the registers and continuously outputs stepper pulses without blocking the image processing pipeline, thereby realizing continuous acquisition, continuous score calculation, and continuous speed / direction adjustment during the continuous movement of the stepper motor.

[0133] On the other hand, embodiments of this application provide an autofocus method based on digital image processing, including:

[0134] Receive pixel streams and generate corresponding pixel coordinates to form a single continuous pixel stream data path;

[0135] In the RAW domain, a pseudo grayscale is constructed by processing the pixel stream, and the pseudo grayscale is then denoised to obtain a denoised pseudo grayscale.

[0136] Five focus windows are fixed, and five paths corresponding to the ROI of each focus window are generated based on pseudo grayscale and pixel coordinates. Gating signal, The value ranges from 1 to 5, and is used in accordance with the above. The gating signal calculates the texture intensity of the five ROIs in the focus window. When the texture intensity of the currently selected ROI is lower than a preset threshold, it switches to the ROI with the highest texture intensity.

[0137] Generate an ROI gating signal based on the currently selected ROI focus window;

[0138] Within the currently selected ROI focus window, three pixel-wise feature values ​​are simultaneously calculated for the denoising pseudo-grayscale of the pixel stream, and then accumulated into three frame-level scores under the ROI gating signal.

[0139] Logarithmically compress and normalize the three frame-level scores to obtain three 8-bit score values. Then, fuse the three 8-bit score values ​​with fixed weights and perform first-order IIR smoothing on the fused result to obtain the final sharpness score.

[0140] Based on the inter-frame variation trend of the final sharpness score, the movement direction and step pulse frequency of the stepper motor are updated on a frame-by-frame basis.

[0141] In one embodiment, the pseudo grayscale satisfies the following: when the pixel coordinates are at the position of a green sampling point, the pseudo grayscale is equal to the pixel coordinates; when the pixel coordinates are at the position of a red or blue sampling point, the pseudo grayscale is the average of the four neighboring green pixels of the pixel coordinates.

[0142] In one embodiment, statistical texture intensity includes:

[0143] Denoising pseudo-grayscale values ​​are convolved with the Sobel operator to obtain the horizontal and vertical gradients;

[0144] The gradient magnitude is obtained by summing the absolute values ​​of the horizontal and vertical gradients.

[0145] A texture indication function is constructed based on the gradient magnitude;

[0146] For each focus window ROI, a texture indication function is used, in Parallel counting of strong edge pixels under gated signals;

[0147] The texture intensity is determined by the ratio of the number of pixels with strong edges to a fixed constant or by the number of pixels with strong edges.

[0148] In one embodiment, the three-path frame-level scores include:

[0149] Score A: Apply the Sobel operator to the denoised pseudo-grayscale to obtain the horizontal and vertical gradients, and accumulate the absolute values ​​of the horizontal and vertical gradients within the pixel clock of the ROI-gated signal to obtain the frame-level gradient energy score; Score B: Calculate the color difference energy of the pixel at the green sampling point and accumulate it within the pixel clock of the ROI-gated signal to obtain the frame-level score; Score C: Calculate the horizontal and vertical Laplacian responses of the denoised pseudo-grayscale, and accumulate the horizontal and vertical Laplacian responses within the pixel clock of the ROI-gated signal to obtain the frame-level improved Laplacian energy and score.

[0150] In one embodiment, the three frame-level scores are logarithmically compressed and normalized, which includes: obtaining the exponent e by detecting the first 1, taking the 6-bit mantissa immediately following the most significant bit as the index, and outputting an 8-bit score value after logarithmically compressing and normalizing by querying a 64×8-bit lookup table to obtain the compensation value.

[0151] In one embodiment, first-order IIR smoothing is performed on the fusion result to obtain the final sharpness score, including:

[0152] The smoothed output is calculated as follows: ,in for Sub-smooth output. for Sub-smooth output. , It is an 8-bit fused value. b=3 is equivalent to α=1 / 8, and a high-pass residual is constructed within the current focus window ROI to calculate the noise intensity index. When the noise intensity index is greater than the set threshold, b is adjusted to 4.

[0153] In one embodiment, updating the movement direction and stepping pulse frequency of the stepper motor 300 on a frame-by-frame basis includes: switching between three states—coarse search, fine search, and lock—on a frame-by-frame basis. The coarse search rapidly sweeps across the depth of field at a high stepping frequency and continuously records the maximum score position pos_max. When the score trend indicates that the peak has been exceeded, the fine search is initiated, and a low-frequency micro-scan is performed near the maximum score position pos_max, while continuously updating the maximum score position pos_max. When the score change enters the stable plateau region, the search returns to the maximum score position pos_max, and the stepping pulse output is stopped to complete the lock.

[0154] To further illustrate the working process and effect of this application in continuous motion autofocus scenarios, several non-limiting examples of the working process are given below. It should be understood that the following examples are only used to illustrate the working mechanism and implementation effect of the technical solution of this application, and do not constitute a limitation on the scope of protection.

[0155] First working example: Basic process of continuous motion focusing: After the device is powered on, the FPGA processor 400 drives the stepper motor 300 to perform a homing action and defines the homing trigger position as pos=0, so as to establish an open-loop position reference for the focusing stroke; then it enters the coarse search state, where the FPGA processor 400 drives the stepper motor 300 to move continuously at a fixed step frequency and continuously acquires the pixel stream at 60fps. At the same time, it calculates the fusion score at the boundary of each frame and maintains the maximum score value Smax and its corresponding maximum score position pos_max, so as to form a "position-score" correspondence sequence during continuous motion; when a continuous downward trend in the fusion score that satisfies the peak crossing criterion is detected, the focusing control state machine 491 switches from coarse search to fine search and reduces the step frequency, so as to perform continuous micro-scans near the maximum score position pos_max and accurately update the maximum score position pos_max; when a change in the fusion score enters the plateau area and satisfies the locking criterion, the focusing control state machine 491 controls the stepper motor 300 to return to the maximum score position pos_max and stops outputting step pulses, thereby completing peak locking and obtaining a stable focusing position.

[0156] Second working example: Parallel statistics of ROIs in multiple focus windows and focus switching: When the image has low texture or the target is not in the center area, the focus window selection and ROI gating circuit is used to generate multiple roi_valid gating signals for a single pixel stream and drive the corresponding frame-level accumulator register to complete the texture intensity statistics of multiple focus window ROIs; when the texture intensity of the default control focus window ROI is lower than the threshold and continues for several frames, the control logic is used to select the focus window with the largest texture intensity as the current focus window and keep it for several frames without switching, so as to avoid frequent focus window jitter causing interference to the continuous closed loop; in the above process, the system always performs gating accumulation statistics based on the same single pixel stream, so as not to output multiple cropped images or copy to form multiple pixel paths.

[0157] Third working example: The effect of denoising and score smoothing on stability.

[0158] In the presence of readout noise, quantization noise, or slight mechanical vibration, the pseudo-grayscale Y... raw (x,y) performs fixed 3×3 kernel denoising to reduce the perturbation of high-frequency noise on the gradient and SML operator (improved Laplacian energy sum) output, so as to suppress score pseudo-peaks and inter-frame jitter; the score processing module is further used to perform first-order IIR smoothing on the fused score, so as to reduce the impact of single-frame fluctuations on the ΔS criterion while maintaining response speed, thereby reducing the risk of false reversal and oscillation during continuous motion focusing and improving peak lock reliability.

[0159] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An autofocus method based on digital image processing, characterized in that, include: Receive pixel streams and generate corresponding pixel coordinates to form a single continuous pixel stream data path; In the RAW domain, a pseudo grayscale is constructed by processing the pixel stream, and the pseudo grayscale is then denoised to obtain a denoised pseudo grayscale. Five focus windows are fixed, and five paths corresponding to the ROI of each focus window are generated based on pseudo grayscale and pixel coordinates. Gating signal, The value ranges from 1 to 5, and is used in accordance with the above. The gating signal calculates the texture intensity of the five ROIs in the focus window. When the texture intensity of the currently selected ROI is lower than a preset threshold, it switches to the ROI with the highest texture intensity. Generate an ROI gating signal based on the currently selected ROI focus window; Within the currently selected ROI (Region of Interest) focus window, three pixel-wise feature values ​​are simultaneously calculated for the denoising pseudo-grayscale of the pixel stream, and these values ​​are accumulated under the ROI gating signal to form three frame-level scores. The three frame-level scores include: Score A: The Sobel operator is applied to the denoised pseudo-grayscale to obtain the horizontal and vertical gradients, and the absolute values ​​of the horizontal and vertical gradients are accumulated within the pixel clock of the ROI gate signal to obtain the frame-level gradient energy score. Score B: Calculate the same-color difference energy of the pixel at the green sampling point and accumulate it within the pixel clock when the ROI gate signal is valid to obtain the frame-level score; Score C: Calculate the horizontal and vertical Laplacian responses of the denoised pseudo-grayscale, and accumulate the horizontal and vertical Laplacian responses within the pixel clock of the ROI-gated signal to obtain the frame-level improved Laplacian energy and score; Logarithmically compress and normalize the three frame-level scores to obtain three 8-bit score values. Then, fuse the three 8-bit score values ​​with fixed weights and perform first-order IIR smoothing on the fused result to obtain the final sharpness score. Based on the inter-frame variation trend of the final sharpness score, the movement direction and step pulse frequency of the stepper motor are updated on a frame-by-frame basis.

2. The autofocus method based on digital image processing according to claim 1, characterized in that, The pseudo grayscale satisfies the following condition: when the pixel coordinates are at the position of the green sampling point, the pseudo grayscale is equal to the pixel coordinates; When the pixel coordinates are at the location of a red or blue sampling point, the pseudo grayscale is the average of the four neighboring green pixels of the pixel coordinates.

3. The autofocus method based on digital image processing according to claim 1, characterized in that, Statistical texture intensity, including: Denoising pseudo-grayscale values ​​are convolved with the Sobel operator to obtain the horizontal and vertical gradients; The gradient magnitude is obtained by summing the absolute values ​​of the horizontal and vertical gradients. A texture indication function is constructed based on the gradient magnitude; For each focus window ROI, a texture indication function is used, in Parallel counting of strong edge pixels under gated signals; The texture intensity is determined by the ratio of the number of pixels with strong edges to a fixed constant or by the number of pixels with strong edges.

4. The autofocus method based on digital image processing according to claim 1, characterized in that, Logarithmic compression and normalization are performed on the three frame-level scores, including: obtaining the exponent e by detecting the first 1, taking the 6-bit mantissa immediately following the most significant bit as the index, and outputting an 8-bit score value after logarithmic compression and normalization by querying a 64×8-bit lookup table to obtain the compensation value.

5. The autofocus method based on digital image processing according to claim 4, characterized in that, The fusion result is subjected to first-order IIR smoothing to obtain the final sharpness score, including: The smoothed output is calculated as follows: ,in for Sub-smooth output. for Sub-smooth output. , It is an 8-bit fused value. b=3 is equivalent to α=1 / 8, and a high-pass residual is constructed within the current focus window ROI to calculate the noise intensity index. When the noise intensity index is greater than the set threshold, b is adjusted to 4.

6. The autofocus method based on digital image processing according to claim 4, characterized in that, The stepper motor's motion direction and step pulse frequency are updated frame-by-frame, including: using three states—coarse search, fine search, and lock—to update and switch between them. The coarse search quickly sweeps across the depth of field at a high step frequency and continuously records the maximum score position pos_max. When the score trend indicates that the peak has been exceeded, the fine search is entered, and a low-frequency micro-scan is performed near the maximum score position pos_max, while continuously updating the maximum score position pos_max. When the score change enters the stable plateau range, the maximum score position pos_max is returned to, and the step pulse output is stopped to complete the lock.

7. An autofocus device based on digital image processing, used to perform the method according to any one of claims 1-6, characterized in that, include: FPGA processor, the FPGA processor being configured to: The image acquisition module is used to receive pixel streams; The data unpacking and line / field reconstruction module is used to generate corresponding pixel coordinates from the pixel stream to form a single continuous pixel stream data path. The pseudo-grayscale construction module is used to process pixel streams in the RAW domain to construct pseudo-grayscale values. A 3x3 denoising filter module is used to denoise the pseudo grayscale to obtain a denoised pseudo grayscale. The focus window selection and ROI gating module is used to fix five focus windows and generate five paths corresponding to the ROI of each focus window based on pseudo grayscale and pixel coordinates. Gating signal, The value ranges from 1 to 5, and is used in accordance with the above. The gating signal calculates the texture intensity of the five ROIs in the focus window. When the texture intensity of the currently selected ROI is lower than a preset threshold, it switches to the ROI with the highest texture intensity. Generate an ROI gating signal based on the currently selected ROI focus window; A sharpness evaluation engine is used to simultaneously calculate three pixel-wise feature values ​​for denoising pseudo-grayscale of the pixel stream within the currently selected focus window ROI, and accumulates them into three frame-level scores under the ROI gating signal. The three frame-level scores include: Score A: The Sobel operator is applied to the denoised pseudo-grayscale to obtain the horizontal and vertical gradients, and the absolute values ​​of the horizontal and vertical gradients are accumulated within the pixel clock of the ROI gate signal to obtain the frame-level gradient energy score. Score B: Calculate the same-color difference energy of the pixel at the green sampling point and accumulate it within the pixel clock when the ROI gate signal is valid to obtain the frame-level score; Score C: Calculate the horizontal and vertical Laplacian responses of the denoised pseudo-grayscale, and accumulate the horizontal and vertical Laplacian responses within the pixel clock of the ROI-gated signal to obtain the frame-level improved Laplacian energy and score; The logarithmic compression and normalization module performs logarithmic compression and normalization on the three frame-level scores to obtain three 8-bit score values. The fixed-weight fusion module is used to fuse the 8-bit scores from the three channels according to fixed weights. The first-order IIR smoothing module is used to perform first-order IIR smoothing on the fusion result to obtain the final sharpness score; The focus and motor control module is used to update the movement direction and step pulse frequency of the stepper motor on a frame-by-frame basis, based on the inter-frame variation trend of the final sharpness score.

8. The autofocus device based on digital image processing according to claim 7, characterized in that, The focusing and motor control module includes: The system includes a stepping pulse generator, a position counter, and a focus control state machine. The stepping pulse generator continuously outputs signals based on the speed register and the direction register to drive the stepping motor. The focus control state machine reads the smooth output at each frame boundary and updates the speed register and the direction register.

Images