An airborne optical imaging intelligent focusing system and method based on deep learning
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
- Filing Date
- 2026-07-08
- Publication Date
- 2026-08-04
AI Technical Summary
鲁棒性差：对光照变化、运动模糊、大气扰动等干扰敏感
[0017] The above-mentioned intelligent focusing system and method for airborne optical imaging based on deep learning has the following effects: it acquires images at different focal lengths to obtain image sequences and performs preprocessing. Based on the preprocessed image sequences, it uses a designed network model to capture the dynamic temporal characteristics of the focusing process according to the sharpness features, realizes the joint prediction of focusing direction, focusing step size and focus confidence, and focuses accordingly. This design makes the focusing path smoother, can achieve a better or even optimal focusing path, has high focusing efficiency, and improves the reliability of the focusing strategy.
Smart Images
Figure CN122513656A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of autofocus technology, and in particular to an airborne intelligent focusing system and method for optical imaging based on deep learning. Background Technology
[0002] With the development of technologies such as unmanned aerial vehicles (UAVs), aerial remote sensing, and airborne reconnaissance, airborne optical imaging systems have increasingly higher requirements for imaging quality in dynamic environments. Autofocus technology is one of the key technologies to ensure image sharpness.
[0003] Traditional autofocus methods primarily rely on image sharpness evaluation functions, such as gradient methods, variance methods, and Laplacian methods, combined with search strategies like hill climbing to achieve autofocus. While these traditional methods perform well in static, texture-rich scenes, in dynamic airborne environments, images are often affected by low contrast, weak texture, motion blur, and changes in lighting, leading to the following problems with autofocus: Low-contrast scene failure: In low-texture areas such as the sky, sea, and desert, the response curve of the traditional sharpness evaluation function is flat and the peak is not obvious. Slow search speed: Traditional hill climbing methods require multiple searches, resulting in slow focusing speed; It is prone to getting trapped in local optima: In complex environments, it is easily disturbed by noise and gets stuck at local extreme points; Poor robustness: sensitive to disturbances such as changes in illumination, motion blur, and atmospheric disturbances.
[0004] In recent years, deep learning technology has been introduced into the field of autofocus, but it suffers from low efficiency and / or poor reliability.
[0005] Therefore, there is an urgent need for a highly efficient and reliable intelligent focusing system and method for airborne optical imaging. Summary of the Invention
[0006] Therefore, it is necessary to provide an airborne optical imaging intelligent focusing system and method based on deep learning to address the above problems.
[0007] To solve the above problems, the present disclosure adopts the following technical solution: In a first aspect, this disclosure provides an airborne optical imaging intelligent focusing system based on deep learning, comprising: The image acquisition module is used to continuously acquire images of the optical system at different focal lengths to obtain an image sequence. The image sequence can reflect the process of image sharpness changing from low to high and then back to low as the focal plane moves. The preprocessing module is used to perform image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module is used to analyze and process the preprocessed image sequence using a sharpness temporal trend-aware autofocus network model, outputting the focusing direction, focusing step size, and focus confidence score. The sharpness temporal trend-aware autofocus network model includes a CNN feature extraction module, a trend perception module, and a focusing decision module. The CNN feature extraction module extracts the sharpness features of each frame in the preprocessed image sequence. The trend perception module captures the contextual dependencies of images during focusing based on the sharpness features, extracting the temporal trend information of sharpness changing with focal length. The focusing decision module includes multiple output heads, and is used to obtain and output the focusing direction, focusing step size, and focus confidence score based on the temporal trend information of sharpness changing with focal length. A focusing control module is used to calculate the focusing control quantity based on the output of the deep learning module. A focusing actuator is used to drive the optical focusing mechanism to operate according to the focusing control quantity.
[0008] In a preferred embodiment, the CNN feature extraction module includes a global average pooling layer and five sequentially arranged convolutional neural network layers. Each convolutional neural network layer is followed by a batch normalization layer and a ReLU activation function layer. The global average pooling layer follows the ReLU activation function layer after the fifth convolutional neural network layer. The convolutional neural network layers are used to extract features related to image sharpness to obtain a feature map. The global average pooling layer is used to reduce the dimensionality of the features and output a feature vector. The feature vector is the sharpness feature of each frame in the preprocessed image sequence.
[0009] In a preferred embodiment, the step of capturing the contextual dependencies of an image during focusing based on sharpness features specifically involves capturing the contextual dependencies of an image during focusing based on sharpness features using a bidirectional gated loop unit.
[0010] In a preferred embodiment, the focusing decision module includes a focusing direction output head, a focusing step size output head, and a focus confidence output head; the focusing direction output head is used to classify the focusing direction using a binary classifier activated by Softmax and output the focusing direction; the focusing step size output head is used to perform focusing step size regression using a linear regression algorithm and output a normalized focusing step size; the focus confidence output head is used to determine and output the focus confidence using a Sigmoid activation function.
[0011] In a preferred embodiment, the loss function of the sharpness temporal trend-aware autofocus network model adopts a multi-task joint loss function, which is: in, Total loss; For direction classification loss; for Weighting coefficients; This is the step size regression loss; for Weighting coefficients; To constrain the loss due to the single-peak trend in sharpness; for Weighting coefficients; The sharpness time-series trend-aware autofocus network model uses the Adam optimizer to update model parameters and minimize the loss function during training.
[0012] In a preferred embodiment, the image acquisition module, preprocessing module, and focusing actuator are implemented based on an FPGA chip, while the deep learning module and focusing control module are implemented based on a single SOC chip.
[0013] In a preferred embodiment, the focusing system further includes a judgment module, which is used to determine whether to terminate focusing based on the focus confidence level and / or the focusing step size.
[0014] Secondly, this disclosure provides a deep learning-based intelligent focusing method for airborne optical imaging, the method being implemented based on the deep learning-based intelligent focusing system for airborne optical imaging described in the first aspect, the method comprising: The image acquisition module continuously acquires images from the optical system at different focal lengths to obtain an image sequence; The preprocessing module performs image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module uses a resolution temporal trend-aware autofocus network model to process the preprocessed image sequence and output the focusing direction, focusing step size, and focus confidence. The focusing control module calculates the focusing control quantity based on the output of the deep learning module. The focusing actuator drives the optical focusing mechanism to operate according to the focusing control quantity.
[0015] In a preferred embodiment, the method further includes: The system monitors focusing time, focusing steps, focusing motor status, and image quality in real time. If any of the monitored parameters are abnormal, focusing is stopped.
[0016] In a preferred embodiment, the focusing includes a coarse adjustment stage and a fine adjustment stage; The focusing method in the coarse adjustment stage includes: The image acquisition module continuously acquires images from the optical system at different focal lengths to obtain an image sequence; The preprocessing module performs image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module uses a resolution temporal trend-aware autofocus network model to process the pre-processed image sequence and output the focusing direction, focusing step size, and focus confidence. The focusing control module calculates the focusing control quantity based on the output of the deep learning module. The focusing actuator drives the optical focusing mechanism to operate according to the focusing control quantity; The focusing method in the fine-tuning stage includes: Based on the sharpness evaluation function, the focusing actuator is used to drive the optical focusing mechanism to achieve focus.
[0017] The above-mentioned intelligent focusing system and method for airborne optical imaging based on deep learning has the following effects: it acquires images at different focal lengths to obtain image sequences and performs preprocessing. Based on the preprocessed image sequences, it uses a designed network model to capture the dynamic temporal characteristics of the focusing process according to the sharpness features, realizes the joint prediction of focusing direction, focusing step size and focus confidence, and focuses accordingly. This design makes the focusing path smoother, can achieve a better or even optimal focusing path, has high focusing efficiency, and improves the reliability of the focusing strategy. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the system structure in one embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a method in one embodiment of the present disclosure; Figure 3 A comparison chart of sharpness-focus position curves; Figure 4 This is a comparison chart of focusing paths; Figure 5 This is a comparison chart of the number of focusing steps. Figure 6 This is a comparison of the final clarity. Detailed Implementation
[0019] The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and preferred embodiments.
[0020] See Figure 1 This embodiment provides an airborne optical imaging intelligent focusing system based on deep learning, including: The image acquisition module is used to continuously acquire images of the optical system at different focal lengths to obtain an image sequence. The image sequence can reflect the process of image sharpness changing from low to high and then back to low as the focal plane moves. The preprocessing module is used to perform image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module is used to analyze and process the preprocessed image sequence using a sharpness temporal trend-aware autofocus network model, outputting the focusing direction, focusing step size, and focus confidence score. The sharpness temporal trend-aware autofocus network model includes a CNN feature extraction module, a trend perception module, and a focusing decision module. The CNN feature extraction module extracts the sharpness features of each frame in the preprocessed image sequence. The trend perception module captures the contextual dependencies of images during focusing based on the sharpness features, extracting the temporal trend information of sharpness changing with focal length. The focusing decision module includes multiple output heads, and is used to obtain and output the focusing direction, focusing step size, and focus confidence score based on the temporal trend information of sharpness changing with focal length. A focusing control module is used to calculate the focusing control quantity based on the output of the deep learning module. A focusing actuator is used to drive the optical focusing mechanism to operate according to the focusing control quantity.
[0021] In this embodiment, the specific structure of the optical focusing mechanism is not limited. Generally, the optical focusing mechanism includes a drive unit, specifically a motor, which can be called a focusing motor. In a specific embodiment, the optical focusing mechanism also includes a focusing lens group. It can be understood that the optical focusing mechanism also includes guiding and transmission components.
[0022] In this embodiment, the focusing actuator is specifically used to drive the focusing motor to move according to the focusing control amount. The driving of the focusing motor can drive the focusing lens group to move, thereby driving the focusing lens group to move.
[0023] It is understandable that driving the optical focusing mechanism to achieve focusing / focusing is used to achieve focus, which means that the ultimate goal is to achieve focus, and does not mean that focus is achieved in one drive.
[0024] In this embodiment, the image acquisition module uses the CameraLink video interface for image acquisition.
[0025] In this embodiment, during the focusing process, images at different focusing positions are acquired by the image acquisition module to obtain an image sequence, denoted as: in, Represents an image sequence. This indicates the first image acquired by the image acquisition module. This indicates the second image acquired by the image acquisition module. Indicates the number of images acquired by the image acquisition module. One image; This image sequence reflects the process of image sharpness changing from blurry to sharp and then from sharp to blurry as the focal plane moves.
[0026] In this embodiment, the image acquisition module is used to transmit the image sequence to the preprocessing module in real time.
[0027] In one embodiment, the airborne optical imaging intelligent focusing system based on deep learning is implemented based on FPGA and SOC. FPGA stands for Field-Programmable Gate Array, and SOC stands for System on a Chip.
[0028] The image acquisition module, preprocessing module, and focusing actuator are implemented based on FPGA chips, while the deep learning module and focusing control module are implemented based on SOC single-chip, meaning the logic is implemented based on chips.
[0029] In one specific embodiment, the FPGA chip can be the domestically produced Fudan Microelectronics JFM7K325T, which is responsible for the following functions: a) Real-time image acquisition via CameraLink interface; b) Image preprocessing, including histogram equalization, Gaussian filtering, image enhancement, etc. c) External sensor data acquisition, such as data acquisition from focusing potentiometer sensors, temperature and other auxiliary information acquisition, etc. d) Focusing motor control: Provide focusing control values to drive the focusing motor to operate.
[0030] In one specific embodiment, the SOC chip may be Rockchip RK3588, which is responsible for the following functions: a) PST-AFNet model inference achieves low-latency computation through hardware accelerators; b) Intelligent focusing system status management and scheduling; c) Focusing strategy decision; d) Data storage and communication, supporting external data interaction.
[0031] This FPGA+SOC architecture fully leverages the advantages of FPGA in real-time signal processing and SOC in complex algorithm inference, enabling parallel processing of image acquisition, preprocessing, model inference, and focus control execution, thus meeting airborne real-time requirements.
[0032] In this embodiment, the image enhancement performed by the preprocessing module specifically refers to enhancing at least one of the image's brightness, saturation, contrast, and outline lines.
[0033] In this embodiment, the preprocessing module eliminates imaging interference caused by atmospheric disturbances, sudden changes in illumination, and sensor noise during airborne imaging, providing high-quality input for the deep learning module.
[0034] In this embodiment, the deep learning module uses a sharpness temporal trend-aware autofocus network model to extract the temporal trend of sharpness change with focal length from the preprocessed image sequence, and determines and outputs the focusing direction, focusing step size and focus confidence based on this. The deep learning module is used to extract the temporal trend of sharpness change with focal length from the image sequence using the PST-AFNet (sharpness temporal trend-aware autofocus network) model, and output the focusing direction, focusing step size and focus confidence. The sharpness temporal trend-aware autofocus network model (PST-AFNet model) includes a CNN feature extraction module, a trend awareness module, and a focus decision module.
[0035] The CNN feature extraction module is used to extract the sharpness features of a single frame image. The trend awareness module is used to capture the contextual dependencies of images in an image sequence based on the sharpness features, and to extract the temporal trend information of sharpness changing with focal length. The focus decision module includes multiple output heads, used to obtain and output the focus direction, focus step size, and focus confidence score based on the temporal trend information of sharpness changing with focal length.
[0036] In this embodiment, the CNN feature extraction module is used to extract the sharpness features (i.e., sharpness-related features) of each single frame image in the preprocessed image sequence. The CNN feature extraction module employs five sequentially arranged convolutional neural network layers. Preferably, the number of channels in the five convolutional neural network layers are 32, 64, 128, 256, and 512, respectively, and the kernel size of each convolutional neural network layer is 3×3. The output of the global average pooling layer is a 512-dimensional feature vector. Each convolutional neural network layer is followed by a BatchNorm (batch normalization) layer and a ReLU activation function layer (understandably, the first convolutional neural network layer is followed by a BatchNorm layer, a ReLU activation function layer, and the second convolutional neural network layer in sequence). Finally, the global average pooling layer maps each frame image into a feature vector. The convolutional neural network layers are used to extract features related to image sharpness to obtain feature maps. The global average pooling layer is used to achieve feature dimensionality reduction; specifically, the input of the global average pooling layer is 512 two-dimensional feature maps, and the output is a 512-dimensional feature vector. The output of the global average pooling layer is the same as the output of the CNN feature extraction module. The global average pooling layer outputs the sharpness features of each frame in the preprocessed image sequence. Through the processing of the CNN feature extraction module, each frame of the image has a feature vector.
[0037] The trend-aware module is used to capture contextual dependencies during the focusing process (i.e., within the image sequence) based on sharpness features using a bidirectional gated cyclic unit (Bi-GRU), with a hidden layer dimension of 256. This module uses continuous... frame( Using feature vectors from images (5~9) as input, the model models the temporal trend of sharpness variation with focal length, and the output can be called temporal fusion feature. Bi-GRU can effectively capture the contextual dependencies in the focusing process, overcoming the limitation of traditional methods that only rely on the information of the current frame.
[0038] The focusing decision module contains multiple output heads, and the outputs of the focusing decision module are: in, Indicates the focusing direction, indicating whether the motor should move forward or backward; Indicates the focusing step size, indicating the distance the motor should move; This represents the confidence level of the focus, indicating the degree of confidence that the deep learning module is about to reach the optimal focus position under the current trend. The focusing decision module includes a focusing direction output head (first output head), a focusing step size output head (second output head), and a focus confidence output head (third output head); The focus direction output head is used for focus direction classification. It employs a Softmax-activated binary classifier to output the focus direction. The focusing direction is used to indicate whether the motor should move in the forward or reverse direction.
[0039] The focusing step size output head is used to perform focusing step size regression. The focusing step size output head adopts a linear regression algorithm to output a normalized focusing step size. The focus confidence output head is used to determine and output the focus confidence level. The focus confidence output head uses a Sigmoid activation function to output the focus confidence level. This focus confidence level represents the degree of confidence that the optimal focus position will be reached.
[0040] The training method for the sharpness temporal trend-aware autofocus network model includes: Build training data; Design the loss function; Design training strategies.
[0041] The specific construction of the training data is as follows: Collect images at different focal plane positions to construct training data: In the training data, each image sequence has a corresponding focus label.
[0042] in, This represents an image sequence, specifically an image sequence processed by the preprocessing module. This indicates the optimal focusing direction and optimal focusing step size. The optimal focusing direction is determined by analyzing the peak position of the sharpness curve, and the optimal focusing step size is quantified based on the distance from the current position of the optical focusing mechanism to the focusing position corresponding to the sharpness peak; the focus label is generated by annotation.
[0043] Specifically, image sequences were collected under various typical airborne scenarios, such as ground, sea, low light, and motion blur, at different focusing positions, and training data was constructed based on these sequences.
[0044] The design of the loss function: This embodiment employs a multi-task joint loss function, specifically including: in, Total loss; For orientation classification loss, cross-entropy loss is used; for Weighting coefficients; For step-size regression loss, Smooth L1 loss is used. for Weighting coefficients; To constrain the loss due to a single-peak trend in sharpness and avoid focusing oscillations; for The weighting coefficients.
[0045] The training strategy is as follows: The Adam optimizer is used to update model parameters and minimize the loss function. Data augmentation strategies are employed during training, including random cropping, contrast adjustment, brightness adjustment, and adding Gaussian noise, to improve the model's generalization ability.
[0046] The sharpness temporal trend-aware autofocus network model predicts focusing parameters by analyzing the temporal trend of sharpness changes with focal length. Its output is: = in, Functions representing deep learning modules; This represents the input to the deep learning model.
[0047] The focus control module uses the focus direction output by the deep learning module. Focusing step size Confidence level of the combined focus This drives the focusing actuator to update the focal plane position. In one embodiment, the rule for updating the focal plane position is: in, For the updated focal plane position, This is the focal plane position before the update.
[0048] In one embodiment, the intelligent focusing system further includes a judgment module for determining whether to terminate focusing based on the focus confidence level and / or the focusing step size.
[0049] If condition one and / or condition two are met, the focusing is terminated and the focusing is completed; otherwise, the focusing continues and the focusing is not completed.
[0050] Condition 1: Focus confidence score output by the sharpness time-series trend-aware autofocus network model If the confidence level exceeds the preset threshold, it indicates that the current position is within the in-focus area; Condition 2: The focusing step size output by the sharpness temporal trend-aware autofocus network model is less than the preset focusing step size threshold. Approaching zero indicates that focusing has converged.
[0051] It is worth noting that the focusing described in this embodiment involves driving the optical focusing mechanism to achieve focus, and does not involve changing the focal length or field of view of the airborne optical system. For example, the optical system is an airborne fixed-focus internal focusing optical system.
[0052] See Figure 2 This paper illustrates a deep learning-based intelligent focusing method for airborne optical imaging, implemented using a deep learning-based intelligent focusing system for airborne optical imaging. The method includes: The image acquisition module continuously acquires images of the optical system at different focal lengths to obtain an image sequence. The image sequence can reflect the process of image sharpness changing from low to high and then back to low as the focal plane moves. The preprocessing module performs image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module uses a resolution temporal trend-aware autofocus network model to analyze and process the preprocessed image sequence, outputting the focusing direction, focusing step size, and focus confidence. The focusing control module calculates the focusing control quantity based on the output of the deep learning module. The focusing actuator drives the optical focusing mechanism to operate according to the focusing control quantity.
[0053] Understandably, repeat the above steps until the optimal focusing position is reached to complete the focusing process.
[0054] In one embodiment, the method includes a step of determining whether to terminate focusing.
[0055] In one specific implementation, the decision to terminate focusing is made based on the focus confidence level and / or the focusing step size.
[0056] In one specific embodiment, focusing is terminated based on focus confidence and / or focusing step size, as well as based on local peaks in sharpness. For example, it is first determined whether focusing has been initially completed based on focus confidence and / or focusing step size (condition one and / or condition two are met to indicate initial completion of focusing, where condition one is the focus confidence output by the sharpness time-series trend-aware autofocus network model). If the preset confidence threshold is exceeded; or if the second condition is that the focusing step size output by the sharpness time-series trend-aware autofocus network model is less than the preset focusing step size threshold, then if the focusing has been initially completed, the optimal focusing position is further determined based on the local peak of sharpness. Based on the optimal focusing position, the focusing actuator drives the focusing motor to adjust the optical system to the optimal focusing position. At this point, focusing is complete and the focusing process is terminated.
[0057] In one embodiment, the method further includes: The system monitors focusing time, focusing steps, focusing motor status, and image quality in real time. If any of the monitored parameters are abnormal, focusing is stopped.
[0058] In one embodiment, the focusing process of the method is divided into a coarse adjustment stage and a fine adjustment stage: In the coarse adjustment stage, the PST-AFNet network model is used to quickly approximate the focus position; The fine-tuning phase uses a sharpness evaluation function for fine adjustments.
[0059] That is, the method is implemented based on the aforementioned airborne optical imaging intelligent focusing system based on deep learning, and the focusing method in the coarse adjustment stage includes: The image acquisition module continuously acquires images from the optical system at different focal lengths to obtain an image sequence; The preprocessing module performs image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module uses a resolution temporal trend-aware autofocus network model to process the pre-processed image sequence and output the focusing direction, focusing step size, and focus confidence. The focusing control module calculates the focusing control quantity based on the output of the deep learning module. The focusing actuator drives the optical focusing mechanism to operate according to the focusing control quantity; The focusing method in the fine-tuning stage includes: Based on the sharpness evaluation function, the focusing actuator is used to drive the optical focusing mechanism to achieve focus.
[0060] Understandably, focusing is achieved through several fine-tuning steps (several steps of "driving the optical focusing mechanism using the focusing actuator based on the sharpness evaluation function").
[0061] The following embodiments describe the online focusing application process of this disclosure in a practical airborne system.
[0062] Step 1, Input image sequence: After the system powers on, the FPGA completes the image acquisition channel configuration, and the SOC loads the PST-AFNet model weights. At this time, the image acquisition module can acquire image sequences in real time and use the current image sequence for coarse focus adjustment.
[0063] Step 2, network outputs focus decision, coarse adjustment stage: Five consecutive frames of images are acquired from the optical system with a relatively large step size to construct an image sequence, which can also be called an image sequence under different focal lengths. After preprocessing, the PST-AFNet model is used to infer and output the focusing direction and step size. The focusing control module calculates the focusing control quantity, and the focusing actuator drives the motor to move rapidly towards the focusing position. The coarse adjustment stage usually only requires 2 to 3 iterations to approach the focusing position, and the focusing time is less than 1 second.
[0064] Step 3, Fine-tuning stage: When the focus confidence score output by the network exceeds 0.8, the system enters the fine-tuning stage. At this time, the traditional sharpness evaluation function is continuously used to search for local peaks to achieve high-precision focus. During the fine-tuning stage, the focus step size is gradually reduced until it is less than the preset focus step size threshold or the focus confidence score exceeds 0.9.
[0065] During the focusing process, the focusing actuator drives the focusing motor to move the focal plane according to the focusing control quantity.
[0066] The focusing process is monitored and anomaly handled. Specifically, the system monitors focusing time, focusing steps, focusing motor status, and image quality in real time. When the following anomalies are detected, an interrupt is triggered and focusing is stopped, while the anomaly status is reported: The focusing time exceeds the preset maximum duration threshold; The number of focusing steps exceeds the mechanical travel limit; The focusing motor is stalled or there is a communication problem. Image quality continues to decline.
[0067] Understandably, this disclosure involves a closed-loop optimization process, continuously repeating predictions, updates, and ultimately convergence. After focusing is complete, the system records the current focus position. When changes in the external environment (such as temperature drift or air pressure changes) trigger refocusing, the system repeats the above process.
[0068] The following example illustrates the differences between this disclosure and prior art.
[0069] The method of combining the coarse adjustment stage with the fine adjustment stage of this disclosure is used to conduct autofocus performance tests with traditional methods (i.e., the traditional methods described in the background art) and existing improved methods (i.e., the sharpness evaluation model based on convolutional neural networks described in the background art). Figure 3 A comparison chart of sharpness-focus position curves; Figure 4 This is a comparison chart of focusing paths. Figure 4 The black area represents the ideal focus curve, which only shows the attenuation portion to the right of the focus peak. Figure 5 This is a comparison chart of focusing steps; the high, medium, and low values on the vertical axis represent the high, medium, and low values of the step count. Figure 6The final sharpness comparison chart shows the high, medium, and low values on the vertical axis. It is evident that this disclosure significantly outperforms existing technologies in terms of focusing speed, accuracy, and robustness.
[0070] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0071] The effects of the airborne optical imaging intelligent focusing system and method based on deep learning disclosed herein are as follows: Image sequences are obtained by acquiring images at different focal lengths and preprocessing them. Based on the preprocessed image sequences, a designed network model is used to capture the dynamic temporal characteristics of the focusing process according to the sharpness features. This enables joint prediction of focusing direction, focusing step size, and focus confidence, and focuses accordingly. This design makes the focusing path smoother, enabling a better or even optimal focusing path with high focusing efficiency and improved reliability of the focusing strategy.
[0072] Specifically: This disclosure employs a joint prediction mechanism of focusing direction, focusing step size, and confidence level: traditional methods typically separate direction determination and step size calculation, leading to inconsistent decisions. The network model of this disclosure is a multi-output neural network that simultaneously outputs the focusing direction, focusing step size, and focusing confidence level, forming an end-to-end focusing decision closed loop. The reliability of the focusing strategy is improved through multi-task joint optimization. Dynamic focusing modeling using multi-frame information: Existing methods mostly rely on single-frame images for sharpness evaluation or depth estimation, neglecting the dynamic temporal characteristics of the focusing process. This disclosure utilizes multi-frame information for dynamic focusing modeling, making the focusing path smoother and more efficient, avoiding repeated oscillations and local optima problems; This disclosure ensures monotonic convergence of the focusing process by constraining the designed loss function, fundamentally solving the problem of focusing path oscillation caused by noise interference in traditional methods such as hill climbing, and improving the stability and reliability of the system.
[0073] A coarse-to-fine focusing strategy: This disclosure organically combines the rapid approximation capability of deep learning with the precise positioning capability of traditional methods. It leverages the robustness of deep learning in complex scenarios while retaining the high-precision characteristics of traditional methods in local search, thus achieving a balance between focusing speed and accuracy.
[0074] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0075] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this patent should be determined by the appended claims.
Claims
1. A deep learning-based airborne optical imaging intelligent focusing system, characterized in that, include: The image acquisition module is used to continuously acquire images of the optical system at different focal lengths to obtain an image sequence. The image sequence can reflect the process of image sharpness changing from low to high and then back to low as the focal plane moves. The preprocessing module is used to perform image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module is used to analyze and process the preprocessed image sequence using a sharpness temporal trend-aware autofocus network model, and output the focusing direction, focusing step size, and focus confidence. The sharpness temporal trend-aware autofocus network model includes a CNN feature extraction module, a trend perception module, and a focusing decision module. The CNN feature extraction module is used to extract the sharpness features of each frame in the preprocessed image sequence. The trend perception module is used to capture the contextual dependencies of the image during the focusing process based on the sharpness features, and extract the temporal trend information of sharpness changing with focal length; the focusing decision module includes multiple output heads, and the focusing decision module is used to obtain and output the focusing direction, focusing step size and focusing confidence based on the temporal trend information of sharpness changing with focal length. A focusing control module is used to calculate the focusing control quantity based on the output of the deep learning module. A focusing actuator is used to drive the optical focusing mechanism to operate according to the focusing control quantity.
2. The airborne optical imaging intelligent focusing system based on deep learning according to claim 1, characterized in that, The CNN feature extraction module includes a global average pooling layer and five sequentially arranged convolutional neural network layers. Each convolutional neural network layer is followed by a batch normalization layer and a ReLU activation function layer. The global average pooling layer follows the ReLU activation function layer after the fifth convolutional neural network layer. The convolutional neural network layers are used to extract features related to image sharpness to obtain feature maps. The global average pooling layer is used to reduce the dimensionality of the features and output feature vectors. The feature vectors are the sharpness features of each frame in the preprocessed image sequence.
3. The airborne optical imaging intelligent focusing system based on deep learning according to claim 2, characterized in that, The specific method of capturing the contextual dependencies of images during focusing based on sharpness features involves using a bidirectional gated loop unit to capture the contextual dependencies of images during focusing based on sharpness features.
4. The airborne optical imaging intelligent focusing system based on deep learning according to claim 2, characterized in that, The focusing decision module includes a focusing direction output head, a focusing step size output head, and a focus confidence output head. The focusing direction output head is used to classify the focusing direction using a binary classifier activated by Softmax and output the focusing direction. The focusing step size output head is used to perform focusing step size regression using a linear regression algorithm and output a normalized focusing step size. The focus confidence output head is used to determine and output the focus confidence using a Sigmoid activation function.
5. The airborne optical imaging intelligent focusing system based on deep learning according to claim 1, characterized in that, The loss function of the sharpness temporal trend-aware autofocus network model adopts a multi-task joint loss function, which is: in, Total loss; For direction classification loss; for Weighting coefficients; This is the step size regression loss; for Weighting coefficients; To constrain the loss due to the single-peak trend in sharpness; for Weighting coefficients; The sharpness time-series trend-aware autofocus network model uses the Adam optimizer to update model parameters and minimize the loss function during training.
6. The airborne optical imaging intelligent focusing system based on deep learning according to claim 1, characterized in that, The image acquisition module, preprocessing module, and focusing actuator are implemented based on FPGA chips, while the deep learning module and focusing control module are implemented based on SOC single-chip.
7. The airborne optical imaging intelligent focusing system based on deep learning according to claim 1, characterized in that, The focusing system also includes a judgment module, which is used to determine whether to terminate focusing based on the focus confidence level and / or the focusing step size.
8. A deep learning-based intelligent focusing method for airborne optical imaging, characterized in that, The method is implemented based on the airborne optical imaging intelligent focusing system based on deep learning as described in any one of claims 1-7, and the method includes: The image acquisition module continuously acquires images from the optical system at different focal lengths to obtain an image sequence; The preprocessing module performs image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module uses a resolution temporal trend-aware autofocus network model to analyze and process the preprocessed image sequence, outputting the focusing direction, focusing step size, and focus confidence. The focusing control module calculates the focusing control quantity based on the output of the deep learning module. The focusing actuator drives the optical focusing mechanism to operate according to the focusing control quantity.
9. The airborne optical imaging intelligent focusing method based on deep learning according to claim 8, characterized in that, The method further includes: The system monitors focusing time, focusing steps, focusing motor status, and image quality in real time. If any of the monitored parameters are abnormal, focusing is stopped.
10. The airborne optical imaging intelligent focusing method based on deep learning according to claim 8, characterized in that, The focusing process includes a coarse adjustment stage and a fine adjustment stage. The focusing method in the coarse adjustment stage includes: The image acquisition module continuously acquires images of the optical system at different focal lengths to obtain an image sequence. The image sequence can reflect the process of image sharpness changing from low to high and then back to low as the focal plane moves. The preprocessing module performs image enhancement, image denoising, region of interest cropping, and image normalization on the images in the image sequence to obtain the preprocessed image sequence; The deep learning module uses a resolution temporal trend-aware autofocus network model to process the pre-processed image sequence and output the focusing direction, focusing step size, and focus confidence. The focusing control module calculates the focusing control quantity based on the output of the deep learning module. The focusing actuator drives the optical focusing mechanism to operate according to the focusing control quantity; The focusing method in the fine-tuning stage includes: Based on the sharpness evaluation function, the focusing actuator drives the optical focusing mechanism to achieve focus.