Automatic focusing method, electronic device, and readable storage medium

Abstract

The present invention provides an automatic focusing method, an electronic device, and a readable storage medium. The automatic focusing method comprises: using a target detection model to detect a target object in an image frame acquired at an initial focal length by an optical imaging device to be focused, so as to acquire a target object detection result; on the basis of the target object detection result, determining whether a target object is present in the image frame; if yes, determining an actual object distance on the basis of the target object detection result and the initial focal length, and on the basis of the actual object distance and a mapping relationship between the object distance and an optimal imaging focal length, determining the optimal imaging focal length; and if not, using a preset search algorithm to search for a focal length until an image having the highest definition value is found, and using a focal length corresponding to the image having the highest definition value as the optimal imaging focal length. The present invention primarily employs a deep learning-based automatic focusing method, supplemented by an image definition evaluation method. Compared with traditional passive focusing methods, the present invention greatly improves the focusing efficiency. Compared with traditional active focusing methods, the present invention eliminates a need for adding an additional optical ranging component, ensuring that the imaging device has a simple structure and a low cost.

Description

Autofocus methods, electronic devices and readable storage media Technical Field

[0001] This invention relates to the field of optical imaging technology, and in particular to an autofocus method, an electronic device, and a readable storage medium. Background Technology

[0002] Traditional imaging devices employ two main autofocus methods: active and passive. Active autofocus is based on optical ranging and requires the imaging device to have an infrared or ultrasonic generator to measure the object distance and calculate the optimal focal length based on optical imaging principles. The advantage of this method is its fast focusing speed, but it requires the imaging device to have ranging capabilities, making the equipment expensive and complex. Passive autofocus, on the other hand, uses image sharpness as the criterion for focusing. This method requires searching the entire zoom range for image sharpness evaluation results using a specific strategy to find the focal length corresponding to the sharpest image. Its advantages include simple and low-cost equipment, but its disadvantages include the need to search the entire zoom range using a specific strategy, resulting in low efficiency, a tendency to get trapped in local optima, and slow focusing speed.

[0003] It should be noted that the information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide an autofocus method, an electronic device, and a readable storage medium. By using a deep learning autofocus method as the main method and an image sharpness evaluation method as a supplement, the autofocus efficiency can be greatly improved compared to the traditional passive autofocus method. Compared to the traditional active autofocus method, no additional optical ranging devices are required, ensuring that the imaging device has a simple structure and low cost.

[0005] To achieve the above objectives, the present invention provides an autofocus method, the autofocus method comprising:

[0006] A pre-trained target detection model is used to detect targets in image frames acquired by the focusing optical imaging device at the initial focal length, so as to obtain the target detection results.

[0007] Based on the target object detection results, it is determined whether a target object exists in the image frame;

[0008] If so, the actual object distance is determined based on the target object detection result and the initial focal length, and the optimal imaging focal length of the optical imaging device to be focused is determined based on the actual object distance and the mapping relationship between the pre-obtained object distance and the optimal imaging focal length.

[0009] If not, a preset search algorithm is used to search for the focal length until the image frame with the highest sharpness value is found, and the focal length corresponding to the image frame with the highest sharpness value is taken as the optimal imaging focal length of the optical imaging device to be focused.

[0010] Optionally, the target detection result includes the target detection box location information, confidence information, and target type information;

[0011] The step of determining whether a target object exists in the image frame based on the target object detection result includes:

[0012] If the confidence level of the target object detection box is greater than or equal to a preset threshold, then it is determined that a target object exists in the image frame;

[0013] If the confidence level of the target object detection box is less than the preset threshold, it is determined that there is no target object in the image frame.

[0014] Optionally, determining the actual object distance based on the target detection result and the initial focal length includes:

[0015] Based on the position information of the target object detection frame, determine the imaging width and imaging height of the target object;

[0016] Based on the target object type information of the target object detection frame, determine the general width and general height of the target object;

[0017] The actual object distance is determined based on the imaging width, imaging height, general width, general height of the target object, and the initial focal length.

[0018] Optionally, determining the actual object distance based on the imaging width, imaging height, general width, general height of the target object and the initial focal length includes:

[0019] The lateral object distance is determined based on the imaging width of the target object and the initial focal length;

[0020] The longitudinal object distance is determined based on the imaging height of the target object and the initial focal length;

[0021] The average of the lateral object distance and the longitudinal object distance is taken as the actual object distance.

[0022] Optionally, the pre-trained object detection model can be obtained through the following steps:

[0023] Obtain a training dataset, which includes multiple training samples, and each training sample includes target object training images and corresponding target object type labels;

[0024] The pre-built object detection model is trained using the training dataset and a pre-set loss function to obtain a trained object detection model.

[0025] Optionally, the pre-set loss function is an improved CIOU_Loss, and the expression of the improved CIOU_Loss is as follows:

[0026] Wherein, IOU represents the overlap between the predicted bounding box and the ground truth bounding box, v represents the similarity between the aspect ratios of the predicted bounding box and the ground truth bounding box, w represents the width of the predicted bounding box, h represents the height of the predicted bounding box, wgt represents the width of the ground truth bounding box, hgt represents the height of the ground truth bounding box, b represents the center point of the predicted bounding box, bgt represents the center point of the ground truth bounding box, ρ represents the Euclidean distance between the center points of the predicted bounding box and the center points of the ground truth bounding box, c represents the diagonal length of the smallest bounding box that simultaneously encloses the predicted bounding box and the ground truth bounding box, α is the weighting coefficient, m represents the average gray value of the pixels in the predicted bounding box, and mgt represents the average gray value of the pixels in the ground truth bounding box.

[0027] Optionally, the target detection model is a YOLO-V4 convolutional neural network model.

[0028] Optionally, for each focal length, the sharpness value of the image frame acquired by the optical imaging device to be focused at any focal length is obtained through the following steps:

[0029] The image frames are subjected to equalization processing to obtain an enhanced image;

[0030] According to a preset image spectrum analysis algorithm, the statistical results of the high-frequency components of the enhanced image are obtained;

[0031] The gradient statistics of the enhanced image are obtained according to the preset two-dimensional Laplacian gradient algorithm.

[0032] The sharpness value of the image frame is obtained based on the statistical results of the high-frequency components and the gradient statistics of the enhanced image.

[0033] Optionally, the step of equalizing the image frame to obtain an enhanced image includes:

[0034] The image frames are converted into YUV image data, and the Y channel image is extracted;

[0035] Adaptive histogram equalization is performed on the Y-channel image to obtain an enhanced image.

[0036] Optionally, obtaining the high-frequency component statistical results of the enhanced image according to a preset image spectrum analysis algorithm includes:

[0037] Based on the wavelet transform algorithm, the horizontal high-frequency coefficients, vertical high-frequency coefficients, and diagonal high-frequency coefficients of each pixel in the enhanced image are obtained.

[0038] Based on the horizontal, vertical, and diagonal high-frequency coefficients of each pixel in the enhanced image, the high-frequency component statistical results of the enhanced image are obtained.

[0039] Optionally, obtaining the gradient statistics of the enhanced image according to a preset two-dimensional Laplacian gradient algorithm includes:

[0040] According to the first preset Laplacian gradient operator, the vertical gradient value of each pixel in the enhanced image is obtained;

[0041] According to the second preset Laplacian gradient operator, the diagonal gradient value of each pixel in the enhanced image is obtained;

[0042] The gradient statistics of the enhanced image are obtained based on the vertical gradient value and diagonal gradient value of each pixel in the enhanced image.

[0043] Optionally, obtaining the sharpness value of the image frame based on the high-frequency component statistical results and gradient statistical results of the enhanced image includes:

[0044] The sharpness value of the image frame is calculated using the following formula: F = aF1 + (1 - aF2)

[0045] Where F is the sharpness value, a is the preset weight value, F1 is the high-frequency component of the enhanced image corresponding to the image frame, and F2 is the gradient value of the enhanced image corresponding to the image frame.

[0046] To achieve the above objectives, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the autofocus method described above.

[0047] To achieve the above objectives, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the autofocus method described above.

[0048] Compared with the prior art, the autofocus method, electronic device, and readable storage medium provided by the present invention have the following advantages:

[0049] The autofocus method provided by this invention first uses a pre-trained target detection model to detect targets in image frames acquired by the optical imaging device to be focused at the initial focal length, thereby obtaining target detection results; then, based on the target detection results, it determines whether a target exists in the image frame; if the determination result is that a target exists, the actual object distance is determined based on the target detection results and the initial focal length, and the optimal imaging focal length of the optical imaging device to be focused is determined based on the actual object distance and the mapping relationship between the pre-obtained object distance and the optimal imaging focal length; if the determination result is not, a preset search algorithm is used to search for the focal length until the image frame with the highest sharpness value is found, and the focal length corresponding to the image frame with the highest sharpness value is taken as the optimal imaging focal length of the optical imaging device to be focused. Therefore, by calculating the actual object distance when a target object is detected, and determining the optimal imaging focal length of the optical imaging device to be focused based on the actual object distance and the pre-acquired mapping relationship between the object distance and the optimal imaging focal length, fast focusing can be achieved. Furthermore, the global nature of constructing the mapping relationship between the object distance and the optimal imaging focal length fundamentally solves the problem of easily getting trapped in local extrema. When no target object is detected, a preset search algorithm is used to search for the focal length, and the optimal imaging focal length of the optical imaging device to be focused is determined based on the focal length corresponding to the image with the highest sharpness value found in the search. This eliminates the need for additional optical ranging devices, ensuring a simple and low-cost imaging device structure. In addition, the autofocus method provided by this invention effectively improves the accuracy of target object detection by using a pre-trained target detection model, effectively avoiding misjudgments and missed detections caused by the variable, complex, and irregular contours of target objects. In summary, the autofocus method provided by this invention, which mainly uses deep learning autofocus and supplements it with image sharpness evaluation method, can greatly improve focusing efficiency compared with traditional passive focusing methods. Compared with traditional active focusing methods, it does not require additional optical ranging devices, ensuring that the imaging device has a simple structure and low cost.

[0050] Since the electronic device and readable storage medium provided by this invention belong to the same inventive concept as the autofocus method provided by this invention, the electronic device and readable storage medium provided by this invention have at least all the beneficial effects of the autofocus method provided by this invention. For details, please refer to the relevant descriptions of the beneficial effects of the autofocus method provided by this invention above. Therefore, the beneficial effects of the electronic device and readable storage medium provided by this invention will not be elaborated here. Attached Figure Description

[0051] Figure 1 is a flowchart illustrating an autofocus method according to an embodiment of the present invention.

[0052] Figure 2 is a schematic diagram of the network structure of the target detection model provided in one embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of the image frame equalization processing provided in an embodiment of the present invention;

[0054] Figure 4 is a comparison of the results of the sharpness evaluation algorithm provided by the present invention with the Laplace sharpness evaluation algorithm and the wavelet transform sharpness evaluation algorithm in the prior art;

[0055] Figure 5 is a block diagram of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0056] The autofocus method, electronic device, and readable storage medium proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of this invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clarify the purpose provided by this invention. Please refer to the drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes and to enable those skilled in the art to understand and read them, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in proportions, or adjustments to the size, provided that the effects and objectives achieved by this invention are the same or similar, should still fall within the scope of the technical content disclosed in this invention.

[0057] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The singular forms “a,” “an,” and “the” include plural objects. The term “or” is generally used to mean “and / or,” the term “several” is generally used to mean “at least one,” and the term “at least two” is generally used to mean “two or more.” Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0058] Furthermore, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0059] The core idea of ​​this invention is to provide an autofocus method, an electronic device, and a readable storage medium. By using a deep learning autofocus method as the main method and an image sharpness evaluation method as a supplement, it can greatly improve focusing efficiency compared to traditional passive focusing methods. Compared to traditional active focusing methods, it does not require additional optical ranging devices, ensuring that the imaging device has a simple structure and low cost.

[0060] It should be noted that the autofocus method provided by this invention can be applied to the electronic device provided by this invention. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, tablet computer, or other hardware device with various operating systems. It should also be noted that, as those skilled in the art will understand, the autofocus method provided by this invention can be used not only for focusing endoscopes but also for focusing other optical imaging devices (such as cameras, microscopes, etc.). This invention does not limit the specific type of optical imaging device to be focused.

[0061] To achieve the above-mentioned ideas, the present invention provides an autofocus method. Please refer to Figure 1, which is a schematic flowchart of an autofocus method provided by an embodiment of the present invention. As shown in Figure 1, the autofocus method provided by the present invention includes the following steps:

[0062] Step S100: Use a pre-trained target detection model to detect the target object in the image frame acquired by the focusing optical imaging device at the initial focal length, so as to obtain the target object detection result.

[0063] Step S200: Based on the target detection result, determine whether there is a target object in the image frame.

[0064] If so, then execute step S300: determine the actual object distance based on the target object detection result and the initial focal length, and determine the optimal imaging focal length of the optical imaging device to be focused based on the actual object distance and the mapping relationship between the pre-obtained object distance and the optimal imaging focal length.

[0065] If not, then proceed to step S400, use a preset search algorithm to search for the focal length until the image frame with the highest sharpness value is found, and take the focal length corresponding to the image frame with the highest sharpness value as the optimal imaging focal length of the optical imaging device to be focused.

[0066] Therefore, by calculating the actual object distance when a target object is detected, and determining the optimal imaging focal length of the optical imaging device to be focused based on the actual object distance and the pre-acquired mapping relationship between the object distance and the optimal imaging focal length, rapid focusing can be achieved (the actual focusing process only has two steps: the first step is focusing to the initial focal length, and the second step is focusing to the optimal imaging focal length). Simultaneously, the global nature of constructing the mapping relationship between the object distance and the optimal imaging focal length fundamentally solves the problem of easily getting trapped in local extrema. Furthermore, by using a preset search algorithm to search for the focal length when no target object is detected, and determining the optimal imaging focal length of the optical imaging device to be focused based on the focal length corresponding to the image with the highest sharpness value found in the search, no additional optical ranging device is needed, ensuring a simple and low-cost imaging device structure. In addition, the autofocus method provided by this invention effectively improves the accuracy of target object detection by using a pre-trained target detection model, effectively avoiding misjudgments and missed detections caused by the variable, complex, and irregular contours of the target object. Furthermore, the autofocus method provided by this invention effectively improves the accuracy of target detection by employing a pre-trained target detection model, thus avoiding misjudgments and missed detections caused by the variable, complex, and irregular contours of target objects. In summary, the autofocus method provided by this invention, which primarily utilizes deep learning-based autofocus and supplements it with image sharpness evaluation methods, significantly improves focusing efficiency compared to traditional passive focusing methods. Compared to traditional active focusing methods, it eliminates the need for additional optical ranging devices, ensuring a simple and low-cost imaging device structure.

[0067] It should be noted that, as those skilled in the art will understand, the initial focal length can be any focusing distance within the focusing range of the optical imaging device to be focused. Preferably, the initial focal length is close to the middle region of the focusing range (i.e., the region close to the middle value of the focusing range). Since the optimal focusing position corresponding to the optimal imaging focal length generally does not appear at the two ends of the focusing range (i.e., the region close to the minimum critical value and the region close to the maximum critical value of the focusing range), by randomly selecting a focusing distance in the middle region of the focusing range as the initial focal length, the sharpness of the image frame acquired by the optical imaging device to be focused at the initial focal length can be effectively guaranteed, thereby effectively ensuring the accuracy of target object detection. Furthermore, it should be noted that, as those skilled in the art will understand, during the process of searching for the focal length using a preset search algorithm, it is necessary to evaluate the image sharpness of the image frames acquired by the optical imaging device to be focused at different focal lengths to obtain the sharpness value of the image frames acquired at each focal length.

[0068] In some exemplary embodiments, the mapping relationship between object distance and optimal imaging focal length can be characterized using an optimal imaging focal length lookup table. Specifically, taking a minimum object distance of 2mm and a maximum object distance of 100mm as an example, an optimal imaging focal length lookup table with 1mm intervals and a length of 98mm is constructed. Each object distance in the table corresponds to an optimal imaging focal length. It should be noted that, as those skilled in the art will understand, the optimal imaging focal length corresponding to each object distance can be obtained through experimental calibration.

[0069] In some exemplary embodiments, the target detection result includes the location information of the target detection box, confidence information, and target type information.

[0070] The step of determining whether a target object exists in the image frame based on the target object detection result includes:

[0071] If the confidence level of the target object detection box is greater than or equal to a preset threshold, then it is determined that a target object exists in the image frame;

[0072] If the confidence level of the target object detection box is less than the preset threshold, it is determined that there is no target object in the image frame.

[0073] Therefore, since the confidence level of the target object detection box represents the probability value that the target object defined by the target object detection box belongs to the detected target object type, the presence of a target object in the image frame can be accurately determined by comparing the confidence level of the target object detection box with a preset threshold. It should be noted that, as those skilled in the art will understand, the specific value of the preset threshold can be set according to actual circumstances, and this invention does not limit it in this regard.

[0074] Another embodiment of target object detection and judgment is as follows: taking the Canny edge extraction of traditional image segmentation algorithm as an example, the target contour is determined after edge extraction by this algorithm, and the color information is used as the basis for target object detection.

[0075] In some exemplary embodiments, determining the actual object distance based on the target detection result and the initial focal length includes:

[0076] Based on the position information of the target object detection frame, determine the imaging width and imaging height of the target object;

[0077] Based on the target object type information of the target object detection frame, determine the general width and general height of the target object;

[0078] The actual object distance is determined based on the imaging width, imaging height, general width, general height of the target object, and the initial focal length.

[0079] In some exemplary embodiments, determining the actual object distance based on the imaging width, imaging height, general width, general height of the target object and the initial focal length includes:

[0080] The lateral object distance is determined based on the imaging width of the target object and the initial focal length;

[0081] The longitudinal object distance is determined based on the imaging height of the target object and the initial focal length;

[0082] The average of the lateral object distance and the longitudinal object distance is taken as the actual object distance.

[0083] Therefore, based on the above focusing and imaging principle formula, we can obtain the following formulas: d1=f*W / w d2=f*H / hd=(d1+d2) / 2

[0084] Wherein, d1 represents the lateral object distance, d2 represents the longitudinal object distance, f represents the initial focal length, W represents the general width of the target object, w represents the imaging width of the target object, H represents the general height of the target object, and h represents the imaging height of the target object.

[0085] In some exemplary embodiments, a target object size database can be pre-built to store the general dimensions (including general width and general height) of different types of target objects. Thus, the general width and general height of the detected target object can be retrieved from the target object database based on the specific type of the detected target object (i.e., the type of target object corresponding to the target object detection frame).

[0086] In some exemplary implementations, a trained object detection model is obtained through the following steps:

[0087] Obtain a training dataset, which includes multiple training samples, and each training sample includes target object training images and corresponding target object type labels;

[0088] The pre-built object detection model is trained using the training dataset and a pre-set loss function to obtain a trained object detection model.

[0089] Specifically, the training images in the training dataset are derived from different types of target object images acquired by optical imaging devices (e.g., endoscopes). By labeling these training images, corresponding target object type labels can be obtained. Further, based on the object distance stored in the optimal imaging focal length lookup table mentioned above, different types of target object images can be acquired at each object distance and stored in the target object data created based on that object distance. Thus, target object training images can be directly obtained from the target object database. It should be noted that, as those skilled in the art will understand, the present invention does not limit the specific type of the target object; the target object can be, but is not limited to, gastrointestinal polyps, inflammatory lesions, etc.

[0090] Furthermore, before labeling the training images, a compilation environment (PyTorch + CUDA + Python) is created and then activated. After labeling the training images, the configuration file can be modified, and then the training images can be enhanced (the specific details of the enhancement process can be found in relevant techniques known to those skilled in the art, and will not be elaborated here). After enhancing the training images, the object detection model can be trained. After training is complete, the model file is output, and the trained object detection model is obtained.

[0091] In some exemplary implementations, the target detection model is a YOLO-V4 convolutional neural network model.

[0092] Specifically, please refer to Figure 2, which is a schematic diagram of the network structure of the target detection model provided in one embodiment of the present invention. As shown in Figure 2, the YOLO-V4 convolutional neural network model (i.e., the target detection model) includes an input end, a backbone network, a neck network, a detection head, and an output end. The backbone network uses the CSPDarkNet-53 network, which has good overall performance in terms of speed and accuracy. Its core employs a Cross-Stage Partial Network. The Cross-Stage Partial Network mainly splits the input features into two parts: one part is passed to the Residual Block for processing, and the other part is directly skipped. The advantage of the Cross-Stage Partial Network structure is that it reduces the computational load while maintaining performance, thereby making more efficient use of feature information. It should be noted that, as those skilled in the art will understand, the specific network structure of the CSPDarkNet-53 network can be found in relevant technologies known to those skilled in the art, and will not be elaborated upon here.

[0093] Furthermore, the Neck Network includes PAN (Path Aggregation Network) and SPP (Spatial Pyramid Pooling Networks). PAN is characterized by fusing features twice, both vertically and horizontally. This operation significantly improves model performance, and the negative impact of its time consumption is offset by the low time consumption of the CSPDarkNet-53 backbone network. SPP addresses the issue of how feature maps of different sizes enter fully connected layers, while directly performing fixed-size pooling on feature maps of arbitrary sizes to obtain a fixed number of features. It should be noted that, as those skilled in the art will understand, the specific network structures of PAN and SPP can be found in relevant technologies known to them, and will not be elaborated upon here.

[0094] Furthermore, the YOLO-V4 convolutional neural network model uses CIOU_Loss as the regression loss function for detection localization, thus optimizing the position loss of the box.

[0095] In some exemplary implementations, the loss function used during the training of the object detection model is a modified CIOU_Loss, the expression of which is as follows:

[0096] Where IOU represents the overlap between the predicted bounding box and the ground truth bounding box, v represents the similarity between the aspect ratios of the predicted and ground truth bounding boxes, w represents the width of the predicted bounding box, and h represents the height of the predicted bounding box. gt h represents the width of the actual bounding box. gt b represents the height of the ground truth bounding box, and b represents the center point of the predicted bounding box. gt Let ρ represent the center point of the ground truth bounding box, ρ represent the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, c represent the diagonal length of the smallest bounding box that simultaneously encloses the predicted and ground truth bounding boxes, α be the weighting coefficient, and m represent the average grayscale value of the pixels in the predicted bounding box. gt This represents the average grayscale value of the pixels within the actual bounding box.

[0097] Therefore, this invention improves the regression loss function of the existing YOLO-V4 convolutional neural network model for detection and localization by using image grayscale as an influencing factor, thus forming the improved CIOU_Loss mentioned above. The improved CIOU_Loss provided by this invention fully considers the degree of overlap between the predicted box and the ground truth box, the distance between the center point, the aspect ratio, and the grayscale difference, thereby making the convergence accuracy of the target detection model higher and the effect more significant.

[0098] In some exemplary embodiments, the search for focal length using a preset search algorithm includes:

[0099] The focal length is searched using a hill-climbing search method.

[0100] Since the hill-climbing search method is highly efficient, focusing efficiency can be further improved by using a preset search algorithm to search for the optimal imaging focal length based on the image sharpness evaluation results.

[0101] In some exemplary embodiments, for each focal length, the sharpness value of the image frame acquired by the focusing optical imaging device at any focal length is obtained through the following steps:

[0102] The image frames are subjected to equalization processing to obtain an enhanced image;

[0103] According to a preset image spectrum analysis algorithm, the statistical results of the high-frequency components of the enhanced image are obtained;

[0104] The gradient statistics of the enhanced image are obtained according to the preset two-dimensional Laplacian gradient algorithm.

[0105] The sharpness value of the image frame is obtained based on the statistical results of the high-frequency components and the gradient statistics of the enhanced image.

[0106] Due to rapid changes in ambient light or when the image sensor in an optical imaging device is in automatic exposure mode, there will be significant differences in brightness between image frames, leading to inaccurate image frame sharpness evaluation results (i.e., inaccurate sharpness values). Therefore, by performing equalization processing on the image frame before calculating its sharpness value to obtain an enhanced image, and then calculating the sharpness value based on the enhanced image corresponding to the image frame, the accuracy of the calculated image frame sharpness value can be effectively guaranteed. This can effectively improve the accuracy of the optimal imaging focal length determined based on the image sharpness evaluation result. Furthermore, since an image is clearer the more high-frequency components it contains, the clarity of an image can be reflected by analyzing and comparing the quantity of high-frequency components. Therefore, by using a preset image spectrum analysis algorithm to obtain the statistical results of the high-frequency components of the enhanced image, and using a preset two-dimensional Laplacian gradient algorithm to obtain the statistical results of the gradient of the enhanced image, and by combining the statistical results of the high-frequency components and the gradient of the enhanced image, the sharpness value of the image frame can be obtained. This allows for a comprehensive evaluation of the image frame's sharpness from both temporal image gradient analysis and frequency domain image high-frequency information analysis, thereby effectively improving the stability of the image sharpness evaluation algorithm and further ensuring the accuracy of the optimal imaging focal length determined based on the image sharpness evaluation results.

[0107] It should be noted that, as those skilled in the art will understand, automatic exposure is achieved by controlling the brightness of the light source and the gain of the image sensor, in order to cope with changes in the external environment and ensure uniform image brightness. However, it is inevitable that this control cannot achieve the effect of minimizing the brightness differences between multiple frames. Furthermore, in non-automatic exposure mode, changes in ambient light will also lead to brightness differences.

[0108] In some exemplary embodiments, the equalization process performed on the image frame to obtain an enhanced image includes:

[0109] The image frames are converted into YUV image data, and the Y channel image is extracted;

[0110] Adaptive histogram equalization is performed on the Y-channel image to obtain an enhanced image.

[0111] Since adaptive histogram equalization changes image contrast by calculating the local histogram of the image and then redistributing brightness, it can effectively improve the local contrast of the image and obtain more image details by performing adaptive histogram equalization on the Y channel image extracted from the image frame. This can further improve the stability of the image sharpness evaluation algorithm and further ensure the accuracy of the optimal imaging focal length determined based on the image sharpness evaluation result.

[0112] Specifically, please refer to Figure 3, which is a schematic flowchart of image frame equalization processing provided by an embodiment of the present invention. As shown in Figure 3, the adaptive histogram equalization processing of the Y-channel image to obtain an enhanced image includes: dividing the Y-channel image into multiple sub-images; performing separate histogram statistics on each sub-image; equalizing each sub-image based on the histogram statistics results; and merging the equalized sub-images using bilinear interpolation to obtain an enhanced image. Therefore, by performing bilinear interpolation when merging the equalized sub-images, the block artifacts of the equalized image data can be effectively eliminated.

[0113] It should be noted that, as those skilled in the art will understand, the specific details of how to equalize sub-images based on histogram statistical results and how to perform bilinear interpolation can be found in the relevant content of adaptive histogram equalization processing flow known to those skilled in the art, and will not be elaborated here.

[0114] In some exemplary embodiments, obtaining the high-frequency component statistical results of the enhanced image according to a preset image spectrum analysis algorithm includes:

[0115] Based on the wavelet transform algorithm, the horizontal high-frequency coefficients, vertical high-frequency coefficients, and diagonal high-frequency coefficients of each pixel in the enhanced image are obtained.

[0116] Based on the horizontal, vertical, and diagonal high-frequency coefficients of each pixel in the enhanced image, the high-frequency component statistical results of the enhanced image are obtained.

[0117] Specifically, the high-frequency component statistics of the enhanced image can be obtained according to the following formula:

[0118] In the formula, F1 represents the statistical results of the high-frequency components of the enhanced image, and c h(i,j) 2 c represents the horizontal high-frequency coefficient of pixel (i,j). v(i,j) 2 c represents the vertical high-frequency coefficient of pixel (i,j). d(i,j)2 This represents the high-frequency focusing coefficient of pixel (i,j).

[0119] It should be noted that, as those skilled in the art will understand, the specific details regarding how to use wavelet transform algorithms to obtain the horizontal, vertical, and diagonal high-frequency coefficients of each pixel in the enhanced image can be found in relevant materials on extracting high-frequency information from images using wavelet transform, which are well-known to those skilled in the art, and will not be elaborated upon here. It should also be noted that, as those skilled in the art will understand, the preset image spectrum analysis algorithm can be, in addition to wavelet transform algorithms, image filtering algorithms or other image spectrum analysis algorithms. For details, please refer to relevant materials on image spectrum analysis, which are well-known to those skilled in the art, and will not be elaborated upon here.

[0120] In some exemplary embodiments, obtaining the gradient statistics of the enhanced image according to a preset two-dimensional Laplacian gradient algorithm includes:

[0121] According to the first preset Laplacian gradient operator, the vertical gradient value of each pixel in the enhanced image is obtained;

[0122] According to the second preset Laplacian gradient operator, the diagonal gradient value of each pixel in the enhanced image is obtained;

[0123] The gradient statistics of the enhanced image are obtained based on the vertical gradient value and diagonal gradient value of each pixel in the enhanced image.

[0124] Therefore, by obtaining the vertical gradient value of each pixel in the enhanced image according to the first preset Laplacian gradient operator, and obtaining the diagonal gradient value of each pixel in the enhanced image according to the second preset Laplacian gradient operator, and obtaining the gradient statistics result of the enhanced image according to the vertical and diagonal gradient values ​​of each pixel in the enhanced image, the problem of the poor noise resistance of the existing Laplacian algorithm causing the image gradient statistics algorithm result to be easily affected by noise can be effectively solved, thereby effectively ensuring the stability of the obtained gradient statistics result.

[0125] Specifically, the first preset Laplacian gradient operator L1 is the Laplacian gradient operator in the prior art, which is a bounding vertical gradient calculation. The second preset Laplacian gradient operator L2 is a bounding diagonal gradient (45° gradient) calculation newly added in this invention. The specific expressions of the first preset Laplacian gradient operator L1 and the second preset Laplacian gradient operator L2 are as follows:

[0126] Furthermore, the vertical gradient value of each pixel in the enhanced image can be calculated according to the following formula:

[0127] Where G1(x,y) represents the vertical gradient value of the pixel at coordinates (x,y) in the image, and f(x,y) represents the pixel value of the pixel at coordinates (x,y) in the image. This represents the convolution operation.

[0128] The diagonal gradient value of each pixel in the enhanced image can be calculated using the following formula:

[0129] G2(x,y) represents the diagonal gradient value (45° gradient value) of the pixel with pixel coordinates (x,y) in the image.

[0130] Furthermore, the gradient statistics of the enhanced image are obtained according to the following formula:

[0131] In the formula, F2 represents the gradient statistics of the enhanced image.

[0132] In some exemplary embodiments, obtaining the sharpness value of the image frame based on the statistical results of the high-frequency components and the gradient statistical results of the enhanced image includes:

[0133] The sharpness value of the image frame is calculated using the following formula: F = aF1 + (1 - aF2)

[0134] Where F is the sharpness value, a is the preset weight value, F1 is the high-frequency component statistical result of the enhanced image corresponding to the image frame, and F2 is the gradient statistical result of the enhanced image corresponding to the image frame.

[0135] It should be noted that, as those skilled in the art will understand, the preset weight value a ranges from (0,1), and the specific value of the preset weight value a can be set according to the actual situation. The present invention does not limit the specific value of the preset weight value a.

[0136] Please refer to Figure 4, which compares the normalized results of the sharpness evaluation algorithm provided by this invention with those of the existing Laplace and wavelet transform sharpness evaluation algorithms. As shown in Figure 4, the existing Laplace sharpness evaluation algorithm (i.e., the original Laplace algorithm in Figure 4) has poor noise resistance, and while the existing wavelet transform sharpness evaluation algorithm (i.e., the wavelet transform in Figure 4) has good noise resistance, its sensitivity is low. In contrast, the sharpness evaluation algorithm provided by this invention (i.e., the algorithm in Figure 4) not only has high sensitivity but also stable performance. It should be noted that, as those skilled in the art will understand, for ease of comparison, the sharpness values ​​in Figure 4 are all normalized results.

[0137] Based on the same inventive concept, the present invention also provides an electronic device. Please refer to Figure 5, which is a block diagram of the electronic device provided in one embodiment of the present invention. As shown in Figure 5, the electronic device provided by the present invention includes a processor 101 and a memory 103. The memory 103 stores a computer program. When the computer program is executed by the processor 101, it implements the autofocus method described above. Since the electronic device provided by the present invention and the autofocus method provided by the present invention belong to the same inventive concept, the electronic device provided by the present invention has at least all the beneficial effects of the autofocus method provided by the present invention. For details, please refer to the relevant descriptions of the beneficial effects of the autofocus method provided by the present invention above. Therefore, the beneficial effects of the electronic device provided by the present invention will not be repeated here.

[0138] As shown in Figure 5, the electronic device further includes a communication interface 102 and a communication bus 104, wherein the processor 101, the communication interface 102, and the memory 103 communicate with each other via the communication bus 104. The communication bus 104 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 104 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 102 is used for communication between the aforementioned electronic device and other devices.

[0139] The processor 101 referred to in this invention can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 101 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines.

[0140] The memory 103 can be used to store the computer program. The processor 101 implements various functions of the electronic device by running or executing the computer program stored in the memory 103 and calling data stored in the memory 103. The memory 103 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable memory (PROM), electrically programmable memory (EPROM), electrically erasable programmable memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, random access memory is available in a variety of forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous random access memory (SDRAM), dual data rate synchronous random access memory (DDRSDRAM), enhanced synchronous random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), memory bus direct random access memory (RDRAM), direct memory bus dynamic random access memory (DRDRAM), and memory bus dynamic random access memory (RDRAM), etc.

[0141] This invention also provides a readable storage medium storing a computer program, which, when executed by a processor, can implement the autofocus method described above. Since the readable storage medium and the autofocus method provided by this invention belong to the same inventive concept, the readable storage medium provides at least all the beneficial effects of the autofocus method provided by this invention. For details, please refer to the relevant descriptions of the beneficial effects of the autofocus method provided by this invention above; therefore, the beneficial effects of the readable storage medium provided by this invention will not be elaborated upon here.

[0142] The readable storage medium provided by this invention can take the form of any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer hard disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.

[0143] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0144] In summary, compared with the prior art, the autofocus method, electronic device, and readable storage medium provided by the present invention have the following beneficial effects:

[0145] By calculating the actual object distance when a target object is detected, and determining the optimal imaging focal length of the optical imaging device to be focused based on the actual object distance and the pre-acquired mapping relationship between the object distance and the optimal imaging focal length, fast focusing can be achieved. Conversely, by using a preset search algorithm to search for the focal length when no target object is detected, and determining the optimal imaging focal length of the optical imaging device to be focused based on the focal length corresponding to the image with the highest search sharpness value, no additional optical ranging devices are needed, ensuring a simple and low-cost imaging device structure. Furthermore, by employing a pre-trained target detection model for target object detection, this invention effectively improves the accuracy of target object detection, effectively avoiding misjudgments and missed detections caused by the variable, complex, and irregular contours of target objects. In summary, this invention, by primarily using a deep learning autofocus method and supplementing it with an image sharpness evaluation method, significantly improves focusing efficiency compared to traditional passive focusing methods. Compared to traditional active focusing methods, it eliminates the need for additional optical ranging devices, ensuring a simple and low-cost imaging device structure.

[0146] It should be noted that computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] It should be noted that the apparatus and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0148] It should also be noted that the above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. An auto-focusing method characterized by, The method comprises the following steps: adopting a pre-trained target detection model to detect a target object in an image frame collected by an optical imaging device to be focused at an initial focal length, so as to obtain a target object detection result; judging whether the image frame contains a target object according to the target object detection result; if yes, determining an actual object distance according to the target object detection result and the initial focal length, and determining an optimal imaging focal length of the optical imaging device to be focused according to the actual object distance and a mapping relationship between an object distance and an optimal imaging focal length obtained in advance; if no, adopting a preset search algorithm to search for a focal length until an image frame with the highest sharpness value is searched, and taking the focal length corresponding to the image frame with the highest sharpness value as the optimal imaging focal length of the optical imaging device to be focused.

2. The autofocusing method of claim 1, wherein, The target object detection result comprises position information, confidence information and target object type information of a target object detection frame; the step of judging whether the image frame contains a target object according to the target object detection result comprises: if the confidence of the target object detection frame is greater than or equal to a preset threshold, it is determined that the image frame contains a target object; if the confidence of the target object detection frame is less than the preset threshold, it is determined that the image frame does not contain a target object.

3. The autofocusing method according to claim 2, wherein The step of determining an actual object distance according to the target object detection result and the initial focal length comprises: determining an imaging width and an imaging height of a target object according to the position information of the target object detection frame; determining a general width and a general height of the target object according to the target object type information of the target object detection frame; determining an actual object distance according to the imaging width, the imaging height, the general width, the general height of the target object and the initial focal length.

4. The autofocusing method according to claim 3, wherein The step of determining an actual object distance according to the imaging width, the imaging height, the general width, the general height of the target object and the initial focal length comprises: determining a horizontal object distance according to the imaging width of the target object and the initial focal length; determining a vertical object distance according to the imaging height of the target object and the initial focal length; taking an average value of the horizontal object distance and the vertical object distance as an actual object distance.

5. The auto-focusing method according to claim 1, wherein The pre-trained target detection model is obtained by the following steps: obtaining a training data set, which comprises multiple training samples, and each training sample comprises a target object training image and a corresponding target object type label; training a pre-constructed target detection model by using the training data set and a pre-set loss function, so as to obtain a trained target detection model.

6. The autofocusing method of claim 5, wherein, The pre-set loss function is an improved CIOU_Loss, and an expression of the improved CIOU_Loss is as follows: wherein, IOU represents a value of the degree of overlap between the prediction box and the real box, v represents the similarity of the height-width ratio of the prediction box and the height-width ratio of the real box, w represents the width of the prediction box, h represents the height of the prediction box, w gt represents the width of the real box, h gt represents the height of the real box, b represents the center point of the prediction box, b gt represents the center point of the real box, p represents the Euclidean distance between the center point of the prediction box and the center point of the real box, c represents the diagonal length of the minimum circumscribed box that simultaneously encloses the prediction box and the real box, a is a weight coefficient, m represents the average gray value of the pixel points in the prediction box, m gt represents the average gray value of the pixel points in the real box.

7. The auto-focusing method according to claim 1, wherein The target detection model is a YOLO-V4 convolutional neural network model.

8. The auto-focusing method according to claim 1, wherein, For each focal length, the sharpness value of an image frame collected by the optical imaging device to be focused at an arbitrary focal length is obtained by the following steps: performing equalization processing on the image frame, so as to obtain an enhanced image; obtaining a high-frequency component statistical result of the enhanced image according to a preset image spectrum analysis algorithm; obtaining a gradient statistical result of the enhanced image according to a preset two-dimensional Laplacian gradient algorithm; obtaining the sharpness value of the image frame according to the high-frequency component statistical result and the gradient statistical result of the enhanced image.

9. The autofocusing method of claim 8, wherein, The image frame is equalized to obtain an enhanced image, including: Converting the image frame into YUV image data and extracting a Y channel image; Adaptive histogram equalization is performed on the Y channel image to obtain an enhanced image.

10. The autofocusing method of claim 8, wherein, The high-frequency component statistical result of the enhanced image is obtained according to a preset image spectrum analysis algorithm, including: According to a wavelet transform algorithm, horizontal high-frequency coefficients, vertical high-frequency coefficients and diagonal high-frequency coefficients of each pixel point in the enhanced image are obtained; According to the horizontal high-frequency coefficients, the vertical high-frequency coefficients and the diagonal high-frequency coefficients of each pixel point in the enhanced image, the high-frequency component statistical result of the enhanced image is obtained.

11. The autofocusing method of claim 8, wherein, The gradient statistical result of the enhanced image is obtained according to a preset two-dimensional Laplace gradient algorithm, including: According to a first preset Laplace gradient operator, a vertical gradient value of each pixel point in the enhanced image is obtained; According to a second preset Laplace gradient operator, a diagonal gradient value of each pixel point in the enhanced image is obtained; According to the vertical gradient value and the diagonal gradient value of each pixel point in the enhanced image, the gradient statistical result of the enhanced image is obtained.

12. The auto-focusing method according to claim 8, wherein, The definition value of the image frame is obtained according to the high-frequency component statistical result and the gradient statistical result of the enhanced image, including: The definition value of the image frame is calculated according to the following formula: F = aF1 + (1-aF2) Wherein, F is the definition value, a is a preset weight value, F1 is the high-frequency component of the enhanced image corresponding to the image frame, and F2 is the gradient value of the enhanced image corresponding to the image frame.

13. An electronic device, comprising: The computer program is stored on the readable storage medium and is executed by the processor to realize the automatic focusing method in any one of claims 1 to 12.

14. A readable storage medium, characterized by, The computer program is stored on the readable storage medium and is executed by the processor to realize the automatic focusing method in any one of claims 1 to 12.

Images