Focusing method and imaging system based on machine learning

Abstract

This invention proposes a focusing method and imaging system based on machine learning. The method includes: capturing a first image by an image capturing device at a first focal length setting, the first focal length setting being used to determine the focal length at which the image capturing device captures the first image; inputting at least a portion of the first image and multiple features into a machine learning model to obtain a predicted focal length setting, wherein these features include the first focal length setting; and controlling the image capturing device to capture a second image based on the predicted focal length setting.

Description

[Technical Field]

[0001] This disclosure relates to a focusing method and imaging system that uses a machine learning model to predict a focal length setting. [Previous Technology]

[0002] In camera technology, in order to obtain a clear image, an automatic focus (AF) mechanism is usually used to adjust the position of the lens elements inside the lens to form a suitable focus. Existing autofocus technologies can be broadly divided into two categories: active focus and passive focus.

[0003] Active autofocus works by emitting infrared, laser, or ultrasonic rays, and a sensor receives the signal reflected from the subject to calculate the object distance, thereby adjusting the focus to achieve focus. This method has the advantages of operating in low-light environments and fast focusing speed because it drives the lens by directly calculating the object distance, without having to try different focal lengths and then judge whether the image is sharp to find the focus. However, active autofocus also has some limitations, such as being easily affected by transparent objects, surface tilt, or environmental interference, which can cause ranging errors. It also requires the integration of transmitting and receiving devices, which limits the miniaturization of the system.

[0004] In contrast, passive autofocus is based on the lens imaging signal. It analyzes image sharpness through algorithms and adjusts the lens position accordingly to achieve focus. The advantage of this method is its adaptability to complex textures and lighting conditions such as backlighting, and the required hardware structure is relatively simple, helping to reduce costs. However, its focusing effect is highly dependent on image quality and algorithm performance, and the algorithms often require iterative calculations, resulting in a focusing speed delay, which is not conducive to focus tracking applications in dynamic scenes. Furthermore, the large amount of computation may increase equipment costs and time burden. Therefore, these methods still have room for improvement. [Summary of the Invention]

[0005] This disclosure proposes a machine learning-based focusing method applicable to an imaging system including an image capturing device. The focusing method includes: capturing a first image by the image capturing device at a first focal length setting value, the first focal length setting value being used to determine the focal length of the image capturing device when capturing the first image; inputting at least a portion of the first image and multiple features into a machine learning model to obtain a predicted focal length setting value, the features including the first focal length setting value; and controlling the image capturing device to capture a second image based on the predicted focal length setting value.

[0006] In one embodiment of this disclosure, the above-mentioned features further include the number of bits of the focal length setting value of the image capturing device, the resolution of the portion of the first image, the shooting distance, and the sharpness of the portion of the first image.

[0007] In one embodiment of this disclosure, the above-described focusing method further includes: calculating a plurality of responses of the portion of the first image based on the Laplacian operator; and calculating sharpness based on these responses.

[0008] In one embodiment of this disclosure, during the training phase, the shooting distance is the distance between the training target object and the image capturing device. The focusing method further includes: during the testing phase, if the relative position between the image capturing device and the test target object is fixed, then the shooting distance is set to the distance between the image capturing device and the test target object; if the relative position between the image capturing device and the test target object is not fixed, the shooting distance is randomly determined.

[0009] In one embodiment of this disclosure, the machine learning model includes a convolutional layer, a flattening layer, a fully connected layer, and an output layer, wherein the output layer is connected to the fully connected layer. The step of inputting the portion of the first image and its features into the machine learning model to obtain a predicted focal length setting includes: inputting the portion of the first image into a convolutional layer to obtain a first feature map; inputting the first feature map into a flattening layer to obtain a first feature vector; concatenating the first feature vector with a second feature vector formed by multiple features to obtain a third feature vector; and inputting the third feature vector into a fully connected layer to obtain a predicted focal length setting from the output layer.

[0010] In one embodiment of this disclosure, the focusing method further includes: adjusting a predicted focal length setting value to generate at least one fine-tuning focal length setting value; controlling an image capturing device to capture at least one third image based on the fine-tuning focal length setting value; and comparing the sharpness of the second image with the sharpness of at least the third image to output the focal length setting value corresponding to the highest sharpness.

[0011] From another perspective, the embodiments disclosed herein provide an imaging system including processing circuitry and an image capturing device for performing the above-described focusing method.

[0012] In the above method and system, the focal length setting value can be calculated quickly, which can greatly reduce the time required for autofocus.

[0013] In order to make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are given in conjunction with the accompanying drawings.

Implementation Method

[0015] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered to be the same or similar components when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the present invention's patent application.

[0016] The terms "first" and "second" used in this document do not specifically refer to order or sequence, but are used only to distinguish elements or operations described with the same technical terms.

[0017] FIG1 is a schematic diagram illustrating an imaging system 100 according to an embodiment. Referring to FIG1, the imaging system 100 includes a processing circuit 110, a memory 120, and an image capturing device 130. The processing circuit 110 is electrically connected to the memory 120 and the image capturing device 130.

[0018] The processing circuit 110 may include a central processing unit, a microprocessor, a microcontroller, a graphics processing unit (GPU), a deep-learning processing unit (DPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The memory 120 may include random access memory, flash memory, or a hard disk, storing multiple instructions and parameters of a machine learning model. The processing circuit 110 executes the instructions in the memory 120 to implement a machine learning-based focusing method. In some embodiments, the memory 120 may also be located within the processing circuit 110.

[0019] The image capturing device 130 is used to sense visible or invisible light to generate an image, and the focal length of the image capturing device 130 is adjustable. In this example, the image capturing device 130 includes a motor 131, a light sensor 132, and a lens group 133. The light sensor 132 may include a charge-coupled device (CCD) sensor, a complementary metal-oxide semiconductor (CMOS) sensor, or other suitable photosensitive element. The lens group 133 may include one or more concave lenses, convex lenses, combined lenses, aspherical lenses, or other types of lenses, and this invention is not limited thereto. The motor 131 is used to receive the focal length setting value calculated by the processing circuit 110, thereby adjusting the position of the lens group 133 and thus changing the focal length. Motor 131 can be a stepper motor or a voice coil motor. When a voice coil motor is used, the focal length setting can be the signal current magnitude, voltage magnitude, or duty cycle. When a stepper motor is used, the focal length setting can be the step count. Those skilled in the art will understand that adjusting the focal length via a motor is a method, which will not be described in detail here. In other embodiments, the image capturing device 130 may also include an electro-optical liquid crystal lens. In these examples, the image capturing device 130 does not include a motor, and the focal length setting is used to control the voltage or electric field, thereby changing the focal length. When the correct focal length is adjusted, optical path 141 is formed, and the image captured by the light sensor 132 is sharp. When the focal length is incorrect, optical path 142 is formed, and the image captured by the light sensor 132 is blurry.

[0020] FIG2 is a flowchart illustrating a focusing method according to an embodiment. Referring to FIG1 and FIG2, in step 201, the image capturing device 130 captures a first image with a first focal length setting value, the first focal length setting value being used to determine the focal length of the image capturing device 130 when capturing the first image. The first image is transmitted to the processing circuit 110. In some embodiments, the first focal length setting value is a preset focal length setting value, or it may be a focal length setting value calculated by other methods.

[0021] In step 202, at least a portion of the first image and multiple features are input to a machine learning model to obtain a predicted focal length setting value, wherein the features at least include the first focal length setting value. In this embodiment, the machine learning model is a neural network, and Figure 3 is a schematic diagram of the neural network architecture according to an embodiment. Referring to Figure 3, this neural network includes one or more convolutional layers 310, flattening layers 320, one or more fully connected layers 330, and an output layer 340. The flattening layer 320 is connected to the fully connected layer 310, and the output layer 340 is connected to the fully connected layer 330. Here, a portion of the first image 302 is taken and input to the convolutional layer 310. This portion may be a region of interest, such as the center of the first image 302. In some embodiments, the location of the user's touch may be detected by an interactive device (e.g., a touch screen), and the region of interest may be defined with the location of the user's touch as the center. In some embodiments, the entire first image 302 may also be input to the convolutional layer 310.

[0022] After passing through the convolutional layer 310, a three-dimensional feature map 312 (also called the first feature map) can be calculated. This first feature map 312 is input to the flattening layer 320 to obtain a feature vector 322 (also called the first feature vector). On the other hand, the above-mentioned multiple features will form a feature vector 304 (also called the second feature vector). The first feature vector 322 and the second feature vector 304 are concatenated to obtain a feature vector 324. The feature vector 324 is input to one or more fully connected layers 330, and finally the output layer 340 obtains the predicted focal length setting value.

[0023] The neural network architecture in Figure 3 is merely an example. In other embodiments, pooling layers, batch normalization layers, or residual layers may be placed between two adjacent convolutional layers 310. In some embodiments, the neural network has only one fully connected layer 330. Alternatively, the third feature vector 324 may be input to a convolutional layer first and then to the fully connected layer 330. This invention does not limit the specific architecture of the neural network. In some embodiments, the machine learning model may also be a random forest, extreme gradient boosting (XGBoost), support vector machine, etc., and this invention is not limited thereto.

[0024] The features included in the feature vector 304 are described here, please refer to Figures 1 and 3. In some embodiments, a total of 5 features are used, namely the number of bits of the focal length setting, the resolution of this part of the first image, the shooting distance, the currently used focal length setting, and the sharpness.

[0025] When different image capturing devices 130 are used, the number of bits for the focal length setting may differ due to the different specifications of the motors 131 within them. For example, some motors have a total of 1024 steps, so the number of bits for the focal length setting is 10; when the number of steps of the motor is 4096, the number of bits for the focal length setting is 12. Different numbers of bits represent different resolutions in which the motor 131 is segmented. The higher the number of bits, the more steps can be segmented, and therefore the physical displacement of each step will be smaller. Treating the number of bits for the focal length setting as a feature in the feature vector 304 can at least partially reflect the specifications of the image capturing device 130.

[0026] The resolution of this part of the first image refers to the number of pixels in the image input to the convolutional layer 310. For example, the resolution of the first image is 1920*1080, and a 400*300 portion (i.e., the region of interest) is extracted from it. Therefore, the value 400*300=120000 is regarded as a feature in the feature vector 304.

[0027] Referring to Figure 1, the shooting distance refers to the distance between the image capturing device 130 and the target object 140, and its unit can be centimeters, meters, or other units. During the training phase, the shooting distance between the target object 140 (also called the training target object) and the image capturing device 130 is known, so this shooting distance can be used as a feature input to the neural network. Multiple training samples at different shooting distances are collected here, so that the neural network can be applied to various shooting distances. During the testing phase (or inference phase), it can be determined whether the relative position between the image capturing device 130 and the target object 140 (at this time called the test target object) is fixed. For example, if the imaging system 100 is applied to a production line to detect whether the product is defective, the relative position between the test target object 140 and the image capturing device 130 is fixed, so the distance between the image capturing device 130 and the test target object 140 can be set as the shooting distance as a feature in the feature vector 304. In other applications, the relative position between the test target object 140 and the image capturing device 130 is not fixed, so the shooting distance can be randomly determined. Although it is randomly determined, it does not reduce the efficiency in some experimental results.

[0028] The first focal length setting value mentioned above will also be used as a feature in the feature vector 304. In this way, the neural network has a reference benchmark, which can improve the accuracy of prediction, that is, reduce the error between the predicted value and the actual value (ground truth).

[0029] Sharpness is the evaluation function used in passive autofocus methods, and in this embodiment, it is calculated based on the Laplacian operator. For example, the Laplacian operator can be represented as a two-dimensional weighted mask. After applying this weighted mask to a portion of the first image, a response can be obtained at each location, thus generating multiple responses. Sharpness can then be calculated based on these responses, for example, by calculating the variance, sum, or other statistical values ​​of these responses. In other embodiments, sharpness can also be calculated using frequency analysis (such as digital cosine transform), gradient square sum, etc., but this invention is not limited to these methods.

[0030] In other embodiments, one or more features in the feature vector 304 may be omitted, for example, the image resolution may be omitted. In other embodiments, other features may also be added to the feature vector 304, such as the aperture size and shutter speed of the image capturing device 130.

[0031] In summary, the feature vector 304 contains information about the image capturing device 130 (number of bits), information about the first image 302 (resolution), relevant settings for capturing the first image (shooting distance and focal length settings), and whether the image is currently at the correct focal length (sharpness). Therefore, the feature vector 304 provides additional information, enabling the neural network to more accurately predict the focal length that will produce a sharp image.

[0032] Referring back to Figure 2, in step 203, the image capturing device 130 is controlled to capture a second image based on the predicted focal length setting value. This predicted focal length setting value is used to determine the focal length of the image capturing device 130 when capturing the second image. This method allows for the determination of the appropriate focal length in one step, improving the speed of autofocus compared to conventional techniques.

[0033] In some embodiments, the predicted focal length setting can be fine-tuned after it is determined. Figure 4 is a schematic diagram illustrating the fine-tuning of the focal length setting according to one embodiment. Referring to Figure 4, the horizontal axis represents the focal length setting when capturing an image, and the vertical axis represents the sharpness of the image. Each coordinate in the figure corresponds to a focal length setting and a sharpness. Coordinate P1 corresponds to the predicted focal length setting F1 and the sharpness C1 of the second image. In some embodiments, one or more (three in this example) fine-tuning focal length settings are generated based on this predicted focal length setting F1, for example, by adding (or subtracting) a certain value to the predicted focal length setting F1 to obtain the fine-tuning focal length setting. Then, the image capturing device 130 is controlled to capture multiple third images based on these fine-tuning focal length settings. For each third image, a corresponding sharpness is calculated. These fine-tuning focal length settings and sharpness correspond to coordinates A1 to A3 in Figure 3. It can be seen that coordinate A2 has the highest sharpness. Next, the sharpness of the second image is compared with the sharpness of these third images to output the focal length setting value corresponding to the highest sharpness. The number of fine-tuning focal length settings mentioned above also represents the number of fine-tuning steps. In other embodiments, the number of fine-tuning focal length settings can be 1, 2, or greater than 3, and this invention is not limited to this. This approach first uses a machine learning model to predict the focal length of the sharp image, and then finds the focal length that produces the highest sharpness within a small range, thus completing the autofocus process.

[0034] Figure 5 is a schematic diagram illustrating the experimental results according to one embodiment. Referring to Figure 5, the first column of Table 500 is the shooting distance, with four options: 10 meters, 1 meter, 30 centimeters, and 12 centimeters. The second column is the time required for autofocus using conventional methods, in seconds. The third column is the time required for autofocus using the method disclosed herein, in seconds, with a fine-tuning step count of 3. As can be seen from Table 500, using the method disclosed herein can significantly reduce the time required for autofocus.

[0035] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0014] Figure 1 is a schematic diagram illustrating an imaging system 100 according to an embodiment. Figure 2 is a flowchart illustrating a focusing method according to an embodiment. Figure 3 is a schematic diagram illustrating the architecture of a neural network according to an embodiment. Figure 4 is a schematic diagram illustrating fine-tuning of the focal length setting value according to an embodiment. Figure 5 is a schematic diagram illustrating experimental results according to an embodiment.

Claims

1. A machine learning-based focusing method applicable to an imaging system, the imaging system including an image capturing device, the focusing method comprising: capturing a first image by the image capturing device at a first focal length setting value, the first focal length setting value being used to determine a focal length at which the image capturing device captures the first image; inputting at least a portion of the first image and a plurality of features into a machine learning model to obtain a predicted focal length setting value, wherein the features include the first focal length setting value; controlling the image capturing device to capture a second image based on the predicted focal length setting value; adjusting the predicted focal length setting value to generate at least one fine-tuned focal length setting value; controlling the image capturing device to capture at least one third image based on the at least one fine-tuned focal length setting value; and comparing the sharpness of the second image and the sharpness of the at least one third image to output a focal length setting value corresponding to the highest sharpness.

2. The focusing method as claimed in claim 1, wherein the features further include one bit of a focal length setting value of the image capturing device, the resolution of the at least portion of the first image, a shooting distance, and the sharpness of the at least portion of the first image.

3. The focusing method as described in claim 2 further includes: Calculate multiple responses of at least a portion of the first image according to a Laplace operator; And calculate the resolution based on those responses.

4. The focusing method as described in claim 2, wherein during a training phase, the shooting distance is the distance between a training target object and the image capturing device, the focusing method further includes: In a testing phase, if the relative position between the image capturing device and the test target object is fixed, the shooting distance is set to the distance between the image capturing device and the test target object; if the relative position between the image capturing device and the test target object is not fixed, the shooting distance is randomly determined.

5. The focusing method as described in claim 1, wherein the machine learning model comprises at least one convolutional layer, a flattening layer, a fully connected layer, and an output layer, wherein the output layer is connected to the fully connected layer, and the step of inputting at least a portion of the first image and the features into the machine learning model to obtain the predicted focal length setting includes: inputting at least a portion of the first image into the at least one convolutional layer to obtain a first feature map; inputting the first feature map into the flattening layer to obtain a first feature vector; concatenating the first feature vector and a second feature vector formed by the features to obtain a third feature vector; and inputting the third feature vector into the fully connected layer to obtain the predicted focal length setting from the output layer.

6. An imaging system comprising: a processing circuit; and an image capturing device electrically connected to the processing circuit, wherein the image capturing device captures a first image at a first focal length setting value, the first focal length setting value being used to determine a focal length at which the image capturing device captures the first image, wherein the processing circuit inputs at least a portion of the first image and a plurality of features into a machine learning model to obtain a predicted focal length setting value, wherein the features include the first focal length setting value, wherein the processing circuit controls the image capturing device to capture a second image based on the predicted focal length setting value; adjusts the predicted focal length setting value to generate at least one fine-tuned focal length setting value; controls the image capturing device to capture at least one third image based on the at least one fine-tuned focal length setting value; and compares the sharpness of the second image and the sharpness of the at least one third image to output a focal length setting value corresponding to the highest sharpness.

7. The imaging system as claimed in claim 6, wherein the features further include one bit of a focal length setting value of the image capturing device, the resolution of the at least portion of the first image, a shooting distance, and the sharpness of the at least portion of the first image.

8. The imaging system of claim 7, wherein the processing circuitry calculates a plurality of responses of the at least portion of the first image based on a Laplacian operator, and calculates the sharpness based on the responses.

9. The imaging system of claim 7, wherein in a training phase, the shooting distance is the distance between a training target object and the image capturing device; in a testing phase, if the relative position between the image capturing device and a test target object is fixed, the processing circuit sets the shooting distance to the distance between the image capturing device and the test target object; if the relative position between the image capturing device and the test target object is not fixed, the processing circuit randomly determines the shooting distance.

10. The imaging system of claim 6, wherein the machine learning model comprises at least one convolutional layer, a flattening layer, a fully connected layer, and an output layer, wherein the output layer is connected to the fully connected layer, and the processing circuitry is configured to: input the at least portion of the first image to the at least one convolutional layer to obtain a first feature map; input the first feature map to the flattening layer to obtain a first feature vector; concatenate the first feature vector with a second feature vector formed by the features to obtain a third feature vector; input the third feature vector to the fully connected layer, and obtain the predicted focal length setting from the output layer.

Images