Automatic focusing method and device, storage medium and computer program product

Abstract

The invention provides an automatic focusing method and device, a storage medium and a computer program product. The method comprises the following steps: an image acquisition part acquires a current image frame; acquiring the definition of the current image frame by using the trained definition evaluation model; determining a state space and an action space of the simulation learning model; the state space comprises the definition, the action space comprises acquisition parameters adopted when the image acquisition part acquires the current image frame, and the acquisition parameters comprise focal length and / or aperture size; and the simulation learning model carries out prediction according to the definition to obtain an optimal acquisition parameter. According to the method, the definition evaluation model is established by using the deep learning technology, and the definition of the image frames under different focal lengths is accurately evaluated through the definition evaluation model. Compared with a traditional heuristic focusing method, the method has better adaptability and generalization ability, and can adapt to focusing requirements under different scenes and shooting conditions. The device can realize the method.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an automatic focusing method and a device thereof, a storage medium, and a computer program product. Background Art

[0002] At present, achieving automatic focusing in fields such as video shooting is an important challenge. Traditional automatic focusing methods are usually based on heuristic rules or image processing technology, and determine the optimal focal length parameters by analyzing the characteristics of the image. However, the performance of these existing methods in complex scenes is often unstable, and they are easily affected by factors such as lighting conditions and background complexity, making it difficult to achieve accurate focusing. In other words, these existing methods do not have good adaptability and generalization capabilities, and cannot meet the needs of video focusing in different scenes and shooting conditions.

[0003] Therefore, it is necessary to improve the deficiencies in the prior art. Summary of the invention

[0004] The main technical problem solved by the present invention is how to improve the adaptability and generalization ability of the auto-focus method. The present invention provides an auto-focus method, which has better adaptability and generalization ability and can meet the focusing requirements under different scenes and shooting conditions.

[0005] According to the first aspect, an embodiment provides an automatic focusing method, including:

[0006] The image acquisition unit acquires the current image frame; and obtains the clarity of the current image frame using the trained clarity assessment model;

[0007] Determine the state space and action space of the imitation learning model; wherein the state space includes the clarity, the action space includes the acquisition parameters used by the image acquisition unit when acquiring the current image frame, and the acquisition parameters include focal length and / or aperture size;

[0008] The imitation learning model predicts the clarity to obtain the optimal acquisition parameters, so as to use the optimal acquisition parameters to acquire the next image frame.

[0009] In one embodiment, the trained clarity assessment model is obtained through the following training process:

[0010] constructing an initial clarity assessment model;

[0011] Acquire a clarity training set; wherein the clarity training set includes a plurality of first image frames and annotations of the first image frames; the annotations are the clarity of the first image frames;

[0012] The initial clarity assessment model is trained using the clarity training set to obtain a trained clarity assessment model.

[0013] In one embodiment, obtaining a clarity training set includes:

[0014] The image acquisition unit acquires images of the first training scene to obtain the plurality of first image frames;

[0015] Calculating the clarity of each of the first image frames, and using the clarity as a label of the first image frame; wherein the acquisition parameters and / or the clarity corresponding to each of the first image frames are different;

[0016] Each of the first image frames and the annotations of the first image frames are used as the clarity training set.

[0017] In one embodiment, the light conditions when each of the first image frames is collected are different; and the first training scene contains moving objects and / or objects that are changing.

[0018] In one embodiment, the imitation learning model is obtained through the following training process:

[0019] The steps of initial training are: performing image acquisition on a second training scene to obtain a plurality of second image frames; using the current imitation learning training set to train the imitation learning model and obtain an imitation learning model that has completed initial training; wherein the imitation learning training set includes the first clarity of the second image frame and the training optimal acquisition parameters manually annotated for the second image frame; the training optimal acquisition parameters include focal length; the first clarity is obtained by the clarity evaluation model;

[0020] The step of preliminary prediction: performing image acquisition on the third training scene to obtain a plurality of third image frames, and obtaining a first definition of the third image frames; the imitation learning model that has completed the initial training performs prediction according to the first definition to obtain a first training optimal acquisition parameter;

[0021] Expert labeling step: manually labeling the third image frame with the second optimal acquisition parameter for training; using the first definition and the second optimal acquisition parameter for training as a new imitation learning training set;

[0022] The step of updating the data set is as follows: obtaining a new imitation learning training set according to the new imitation learning training set;

[0023] The step of stopping iteration: repeating the steps of the initial training to the step of updating the data set until the condition of stopping iterative training is reached, and using the imitation learning model at this time as the imitation learning model of the completed training.

[0024] In one embodiment, the initial training step also includes: determining a reward function of the imitation learning model; the preliminary prediction step also includes: the image acquisition unit completes automatic focusing according to the first training optimal acquisition parameters, and again acquires the image of the third training scene to obtain a new third image frame; obtains a second clarity of the new third image frame; wherein the reward function is obtained based on the difference between the above-mentioned second clarity and the first clarity; the condition for stopping iterative training is: the value of the reward function reaches a preset reward value threshold.

[0025] According to the second aspect, an embodiment provides an automatic focusing device, comprising:

[0026] An image acquisition unit, configured to acquire a current image frame;

[0027] A definition acquisition unit is configured to acquire the definition of the current image frame using the trained definition evaluation model;

[0028] The optimal strategy output unit is configured to determine the state space and action space of the imitation learning model; wherein the state space includes the clarity, and the action space includes the acquisition parameters of the image acquisition unit when acquiring the current image frame, and the acquisition parameters include focal length and / or aperture size; the trained imitation learning model predicts based on the clarity to obtain the optimal acquisition parameters; wherein the image acquisition unit can complete automatic focusing based on the optimal acquisition parameters.

[0029] In one embodiment, the trained imitation learning model is obtained through the following training process:

[0030] The steps of initial training: the image acquisition unit acquires images of the second training scene to obtain a plurality of second image frames; the imitation learning model is trained using the current imitation learning training set to obtain an imitation learning model that has completed initial training; wherein the imitation learning training set includes the first clarity of the second image frame and the training optimal acquisition parameters manually annotated for the second image frame; the training optimal acquisition parameters include focal length and / or aperture size; the first clarity is obtained by the clarity evaluation model;

[0031] The step of preliminary prediction: the image acquisition unit acquires images of the third training scene to obtain a plurality of third image frames, and obtains a first definition of the third image frames; the imitation learning model that has completed the initial training performs prediction according to the first definition to obtain a first training optimal acquisition parameter;

[0032] Expert labeling step: manually labeling the third image frame with the second optimal acquisition parameter for training; using the first definition and the second optimal acquisition parameter for training of the third image frame as a new imitation learning training set;

[0033] The step of updating the data set is as follows: obtaining a new imitation learning training set according to the new imitation learning training set;

[0034] The step of stopping iteration: repeating the steps of the initial training to the step of updating the data set until the condition of stopping iterative training is reached, and using the imitation learning model at this time as the imitation learning model of the completed training.

[0035] According to a third aspect, an embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium includes a program, wherein the program can be executed by a processor to implement a method as described in any embodiment of the present invention.

[0036] According to a fourth aspect, in one embodiment, a computer program product is provided, comprising a computer program and / or instructions, wherein when the computer program and / or instructions are executed by a processor, the method described in any one of the embodiments herein is implemented.

[0037] The beneficial effects of this application are:

[0038] The method firstly acquires the current image frame by the image acquisition unit; obtains the clarity of the current image frame by using the trained clarity evaluation model; determines the state space and action space of the imitation learning model; the state space includes the above-mentioned clarity, and the action space includes the acquisition parameters adopted by the image acquisition unit when acquiring the current image frame, and the acquisition parameters include focal length and / or aperture size; the imitation learning model predicts the optimal acquisition parameters according to the clarity, and adopts the optimal acquisition parameters to acquire the next image frame; since the imitation learning model is obtained by training with the labeled data set, that is, by imitating the expert operation, the imitation learning model learns the clarity change law of the image frame sequence at different focal lengths, therefore, the imitation learning model can predict the optimal acquisition parameters according to the clarity of the current image frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart of an automatic focusing method according to an embodiment;

[0040] Figure 2 A schematic diagram of a process for training a clarity assessment model according to an embodiment;

[0041] Figure 3 A schematic diagram of a process of obtaining a clarity training set according to an embodiment;

[0042] Figure 4A schematic diagram of a process for training an imitation learning model according to an embodiment;

[0043] Figure 5 A schematic diagram of a module of an autofocus device according to an embodiment. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0045] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0046] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0047] In recent years, with the development of deep learning technology, deep learning-based autofocus methods have gradually attracted attention. These methods use deep neural network models to learn the clarity characteristics of video frames at different focal lengths from a large amount of data (such as video frames) to achieve more accurate focusing.

[0048] The technical concept of this autofocus method is: combining deep learning technology and imitation learning (a machine learning method) to achieve accurate learning of the law of changes in the clarity of image frames at different focal lengths. Specifically, this autofocus method first uses deep learning technology to establish a clarity evaluation model; the clarity corresponding to the image frame (such as video frame) sequence at different focal lengths is obtained through the clarity evaluation model; then, the imitation learning model is trained using the labeled data set, that is, the imitation learning model learns the law of changes in the clarity of the image frame sequence at different focal lengths by imitating the expert's operation.

[0049] The technical solution of the present application will be described in detail below in conjunction with embodiments.

[0050] Please refer to Figure 1 , the present application provides an automatic focusing method. The automatic focusing method comprises:

[0051] Step S100: capturing a current image frame; obtaining the clarity of the current image frame;

[0052] Step S200: determining the state space and action space of the imitation learning model;

[0053] Step S300: Predict according to the clarity to obtain the optimal acquisition parameters.

[0054] Specifically, step S100 includes: the image acquisition unit acquires the current image frame; and obtains the clarity of the current image frame using the trained clarity assessment model.

[0055] In some embodiments, the current image frame and the acquisition parameters used by the image acquisition unit when acquiring the current image frame can be input into a trained clarity evaluation model, and then the clarity of the current image frame can be obtained using the trained clarity evaluation model. The above acquisition parameters can include focal length.

[0056] In some embodiments, the acquisition parameters may include focal length and / or aperture size.

[0057] Specifically, in step S200, the state space includes the above-mentioned clarity, and the action space includes the acquisition parameters used by the image acquisition unit when acquiring the current image frame, and the acquisition parameters include the focal length.

[0058] In some embodiments, the state space may include the current image frame and the definition.

[0059] In some embodiments, the state space may include the current image frame, the definition, and acquisition parameters used by the image acquisition unit when acquiring the current image frame.

[0060] Specifically, step S300: the trained imitation learning model predicts according to the clarity to obtain the optimal acquisition parameters, so as to use the optimal acquisition parameters to acquire the next image frame; wherein the image acquisition unit can complete automatic focusing according to the optimal acquisition parameters.

[0061] In some embodiments, step S300 may also include: the trained imitation learning model predicts the optimal acquisition parameters according to the clarity and the acquisition parameters used by the image acquisition unit when acquiring the current image frame, so as to acquire the next image frame using the optimal acquisition parameters. In some embodiments, in step S100, the image acquisition unit may be a camera or a webcam, etc. The specific composition of the image acquisition unit is not limited here. The image acquisition unit may be set on a robot for fast movement.

[0062] In some embodiments, the clarity assessment model is mainly used to assess the clarity characteristics of the image frame. Figure 2 , the trained clarity assessment model is obtained through the following training process:

[0063] Step S110: constructing an initial clarity assessment model;

[0064] Step S120: obtaining a definition training set; wherein the definition training set includes a plurality of first image frames and annotations of the first image frames; the annotations are the definition of the first image frames;

[0065] Step S130: The initial clarity assessment model is trained using the clarity training set to obtain a trained clarity assessment model.

[0066] In some embodiments, in step S110, the clarity evaluation model may include a convolution layer, a pooling layer, a BN layer, an activation function, etc. The BN layer standardizes each batch of training data, that is, uses the mean and variance of each batch of data.

[0067] In some embodiments, in step S110, the clarity assessment model may also directly adopt an existing deep learning model, that is, the pre-trained deep learning model may be directly fine-tuned to obtain the required clarity assessment model. For example, the clarity assessment model may adopt an existing ResNet network (a deep neural network structure).

[0068] It should be noted that, in step S110, the specific structure of the definition evaluation model is not limited. Any network model that can be used to evaluate the definition features of an image frame can be used as the initial definition evaluation model or the definition evaluation model.

[0069] For some examples, please refer to Figure 3 In step S120, obtaining a clarity training set includes:

[0070] Step S121: the image acquisition unit acquires images of the first training scene to obtain a plurality of first image frames;

[0071] Step S122: calculating the clarity of each first image frame, and using the clarity as a label of the first image frame;

[0072] Step S123: taking each first image frame and the annotation of the first image frame as a clarity training set.

[0073] In some embodiments, in step S121, the image acquisition unit may acquire images of the same or different first training scenes to obtain multiple first image frames. Those skilled in the art may determine the first training scene according to actual needs, and the first training scene is not limited here.

[0074] It should be noted that in step S122, the clarity of each first image frame is calculated using the existing technology, so the specific calculation process of the clarity is not repeated here. The acquisition parameters and / or clarity corresponding to each first image frame are different.

[0075] In some embodiments, in step S122, the focal lengths of the plurality of first image frames can cover all focal lengths of the image acquisition unit. Preferably, the clarity of any two first image frames is different.

[0076] In some embodiments, in step S122, acquisition parameters of any two first image frames are different.

[0077] In some embodiments, in step S122, the clarity of any two first image frames is different.

[0078] In some embodiments, in step S122, the acquisition parameters and sharpness of any two first image frames are different.

[0079] In some embodiments, the light conditions when each first image frame is collected are different; the scene contains moving objects and / or objects that are changing. Since the samples in the definition training set contain factors such as different definition levels, different focal lengths, different light conditions, close-ups, distant views, and object motion and changes at different focal lengths, the generalization and robustness of the method can be improved.

[0080] In some embodiments, in step S123, each first image frame, the acquisition parameters used by the image acquisition unit when acquiring the first image frame, and the annotations of the first image frame can be used as a clarity training set. In other words, the clarity training set can include each first image frame, the acquisition parameters used by the image acquisition unit when acquiring the first image frame, and the annotations of the first image frame.

[0081] It should be noted that the specific process of "training the initial clarity assessment model using the clarity training set to obtain a trained clarity assessment model" in step S130 belongs to the prior art in the field, so it will not be repeated here.

[0082] It can be seen that in some embodiments, image frame samples with different focal lengths and sharpness are collected to construct a clarity training set, and then the clarity assessment model is trained through the clarity training set, so that the clarity assessment model can learn or understand the dynamic changes between focus position and clarity.

[0083] In some embodiments, the imitation learning model is used to predict the optimal acquisition parameters (such as the optimal focal length) based on the clarity obtained in step S100. The above clarity is used to evaluate the quality of the current image frame. The imitation learning used in this imitation learning model is to minimize the action difference between the agent strategy and the expert strategy. In order to increase the robustness of the imitation learning strategy, new data sets are added to the optimization strategy to assist in optimization training. Please refer to Figure 4 , the above-mentioned trained imitation learning model can be obtained through the following training process:

[0084] Initial training step S210: collecting images of a second training scene to obtain a plurality of second image frames; using the current imitation learning training set to train the imitation learning model and obtain an imitation learning model that has completed initial training;

[0085] The preliminary prediction step S220: performing image acquisition on the third training scene to obtain a plurality of third image frames, and obtaining a first definition of the third image frames; the imitation learning model that has completed the initial training performs prediction according to the first definition to obtain a first training optimal acquisition parameter;

[0086] Expert labeling step S230: manually labeling the third image frame with the second optimal acquisition parameter for training; using the first definition and the second optimal acquisition parameter for training of the third image frame as a new imitation learning training set;

[0087] Step S240 of updating the data set: obtaining a new imitation learning training set according to the new imitation learning training set;

[0088] Step S250 of stopping iteration: repeating steps S210 of initial training to step S240 of updating the data set until the condition of stopping iterative training is reached, and the imitation learning model at this time is regarded as the imitation learning model of completed training.

[0089] It should be noted that the imitation learning model in steps S200 and S300 is obtained based on the imitation learning technology. The purpose of imitation learning and reinforcement learning is the same: the purpose of both is to learn the policy network and thus control the intelligent agent. However, the principles of imitation learning and reinforcement learning are different: imitation learning is to learn from human experts, and the goal is to make the decision made by the policy network the same as that of human experts; while reinforcement learning is to improve the strategy using the rewards of environmental feedback, and the goal is to maximize the cumulative reward (i.e., return). There are two main methods of imitation learning: behavior cloning and inverse reinforcement learning.

[0090] It should be noted that in step S200, a state in the state space is used as a description of a certain moment (such as a current image frame), and an action in the action space represents the behavior taken by the agent in a certain state (such as the acquisition parameters or optimal acquisition parameters that should be selected for the current image frame).

[0091] In some embodiments, in step S210, the image acquisition unit may acquire a plurality of second image frames by acquiring images of the second training scene. The imitation learning training set includes the first clarity of the second image frame and the first training optimal acquisition parameter manually annotated for the second image frame; the training optimal acquisition parameter (such as the first training optimal acquisition parameter) includes the focal length.

[0092] In some embodiments, in step S210, the imitation learning training set may also include acquisition parameters used by the image acquisition unit when acquiring the second image frame. In other words, the imitation learning training set may include the acquisition parameters used by the image acquisition unit when acquiring the second image frame, the first clarity of the second image frame, and the first training optimal acquisition parameters manually annotated for the second image frame.

[0093] It should be noted that, in step S210, those skilled in the art can determine the second training scene by themselves according to actual needs, and the second training scene is not limited here. The second training scene may be the same as the first training scene, or the second training scene may be different from the first training scene. For example, the "several second image frames" in the imitation learning training set may include part or all of the first image frames in the clarity training set. Those skilled in the art can determine the specific composition of the imitation learning training set by themselves according to actual needs. Experts in this field can manually annotate the second image frame with the optimal acquisition parameters (such as the optimal focal length). If a new second image frame is obtained by acquiring the optimal acquisition parameters for training, the clarity corresponding to the new second image frame can exceed the preset clarity threshold. The preset clarity threshold can be set by those skilled in the art.

[0094] It should be noted that, in step S210, the acquisition parameters corresponding to each second image frame are different; the imitation learning model that has completed the initial training can learn the initial focus strategy. "Focus strategy" refers to the rule or function for the agent to select actions in a given state, that is, the rule or function for selecting the corresponding optimal acquisition parameters for an image frame with a known clarity. For example, if the clarity of the current image frame is not suitable for the current focal length setting, the focus strategy will predict a new optimal acquisition parameter (such as focal length), such as 35mm, to obtain a current image frame with higher clarity.

[0095] In some embodiments, in step S210, the first definition of the second image frame and the acquisition parameters used by the image acquisition unit when acquiring the second image frame may be input into the imitation learning model, and the imitation learning model may be trained.

[0096] In some embodiments, in step S210, only the first definition of the second image frame may be input into the imitation learning model, and the imitation learning model may be trained.

[0097] It should be noted that the specific process of step S210, "using the current imitation learning training set to train the imitation learning model and obtain the imitation learning model that has completed the initial training" belongs to the prior art in this field, so it will not be described in detail here.

[0098] It should be noted that both step S210 and step S220 need to calculate the corresponding reward function.

[0099] In some embodiments, in step S210 and step S220, the first clarity is obtained by a trained clarity assessment model.

[0100] In some embodiments, in step S220, the third training scene is different from the second training scene. The image acquisition unit may acquire images of the third training scene to obtain a plurality of third image frames.

[0101] It should be noted that, in step S220, those skilled in the art can use the image acquisition unit to acquire the third image frame based on the training optimal acquisition parameters according to actual needs. The third image frame is not specifically limited here. The acquisition parameters corresponding to each third image frame are different.

[0102] It should be noted that in step S230, preferably, an expert in the field can manually mark the state (such as clarity) of the third image frame with an action (such as the second optimal acquisition parameter for training). The role of "manually marking the second optimal acquisition parameter for training the third image frame" is to achieve strategy optimization faster by introducing additional information and using expert demonstrations with tendencies, that is, using the innate ability of humans and animals to imitate the behavior of other individuals, and learning strategies by imitating the behavior of other individuals, thereby accelerating the learning speed and improving performance. If a new third image frame is obtained by acquiring the optimal acquisition parameters for training, the second clarity corresponding to the new third image frame, that is, the newly obtained clarity after automatic focusing, can exceed the preset clarity threshold or be better than the first clarity of the original third image frame. Personnel in the field can determine the preset clarity threshold according to actual needs. Among them, the proportion of each type of sample in the imitation learning training set and the new imitation learning training set should be consistent, thereby avoiding overfitting to a specific type of data.

[0103] In some embodiments, in step S240, obtaining a new imitation learning training set according to the new imitation learning training set includes: merging the new imitation learning training set into the imitation learning training set to obtain a new imitation learning training set.

[0104] In some embodiments, in step S240, obtaining a new imitation learning training set according to the new imitation learning training set includes: using the new imitation learning training set as a new imitation learning training set.

[0105] It should be noted that the role of step S240 is to update the imitation learning training set in step S210, thereby increasing the diversity of the updated imitation learning training set, such as adding image frames with different lighting conditions and different scene details, etc. The above diversity helps the imitation learning model to obtain better generalization performance. In other words, the above steps S210 to S250 are a training method for iterative learning, which allows the imitation learning model to learn from its own experience, and this experience usually includes those situations that may not be covered in the original training data set (such as the imitation learning training set before the update), thereby improving the robustness of the method.

[0106] In some embodiments, in step S250, the condition for stopping iterative training may be: the number of iterations of repeating the initial training step S210 to the step S240 of updating the data set reaches a preset number threshold. Those skilled in the art may determine the preset number threshold according to actual needs.

[0107] It should be noted that the imitation learning model can also be directly obtained by using the existing training process. In some embodiments, the imitation learning model of the present application is different from the prior art in that: during feature extraction, the trained clarity assessment model is used to extract features from the input current image frame, thereby obtaining the clarity of the current image frame. In some embodiments, the imitation learning model adopts an online iterative training process (such as steps S210 to S250 above), that is, the performance of the imitation learning model is gradually improved through continuous feedback and updating of the data set. This is different from a one-time training process, which usually does not involve continuous updating of the data set.

[0108] In some embodiments, the initial training step S210 also includes: determining a reward function of the imitation learning model; the preliminary prediction step S220 also includes: the image acquisition unit completes automatic focusing according to the optimal acquisition parameters of the first training, and again acquires an image of the third training scene to obtain a new third image frame; obtains a second clarity of the new third image frame; wherein the reward function is obtained based on the difference between the second clarity and the first clarity; the condition for stopping iterative training is: the value of the reward function reaches a preset reward value threshold.

[0109] In some embodiments, those skilled in the art can determine the specific form of the reward function according to actual needs. For example, the reward function may include a second clarity (i.e., the newly obtained clarity after automatic focusing). The reward function may also include the difference between the second clarity and the first clarity. For example, when the clarity evaluation value increases (such as the above difference is greater than zero), the value of the current reward function is positive, that is, the current imitation learning model obtains a positive reward; when the clarity evaluation value decreases (such as the above difference is less than zero), the value of the current reward function is negative or zero, that is, the current imitation learning model obtains a negative reward or no reward. The reward function here is used to guide the imitation learning model to learn how to predict the optimal acquisition parameters (such as the optimal focal length) based on the clarity characteristics of the current image frame. Among them, the functions of the above second clarity, the first training optimal acquisition parameters and / or the second training optimal acquisition parameters include: guiding the imitation learning model to find the optimal acquisition parameters (such as the optimal focal length) according to the current clarity, that is, the above second clarity is used as feedback to the imitation learning model to guide the imitation learning model to judge the pros and cons of the current acquisition parameters (such as focal length).

[0110] In some embodiments, the reward function mainly considers two factors: the clarity of the image (such as the first clarity and the second clarity) and the adjustment range of the acquisition parameters. The reward function is used to guide the intelligent agent (such as the imitation learning model) to adopt a more reasonable focusing strategy. For example, the expression of the reward function can be:

[0111] R = α·C-β·|P|;

[0112] Among them, C represents the clarity of the image; |P| represents the sum of the changes in the above acquisition parameters in two consecutive acquisitions of the current image; α represents the first weight coefficient, α>0; β represents the second weight coefficient, β>0. It can be seen from the expression of the reward function that: when the value of C is larger, that is, the image clarity of the current image is higher, the value of the reward function is larger; when the value of |P| is smaller, that is, the absolute value of the above sum is smaller, the value of the reward function is larger. The above |P| is determined according to the adjustment range of the camera parameters (such as the above acquisition parameters).

[0113] In some embodiments, the reward function mainly considers three factors: image clarity, the adjustment range of acquisition parameters, and whether the image clarity reaches a preset image clarity threshold. For example, the expression of the reward function is:

[0114] R = α·C-β·|P|+γ·G;

[0115] Where C represents the image clarity; |P| represents the sum of the absolute values ​​of the above changes; G is equal to 1 or 0, 1 means that the image clarity reaches the preset image clarity threshold C goal , 0 means the image clarity does not reach the preset image clarity threshold C goal ; α represents the first weight coefficient; β represents the second weight coefficient; γ represents the third weight coefficient, γ>α>β>0. It can be understood that the above γ·G represents the deep reinforcement learning model because the image clarity reaches the preset image clarity threshold C goal The extra reward obtained. Usually γ is a large positive number. Preferably, γ>>α>β. Those skilled in the art can determine α, β, γ and C by themselves. goal For example, α = 10, β = 0.1, γ = 100, C goal =0.8. It can be seen from the expression of the reward function that: when the value of C is larger, that is, the image clarity of the current image is higher, the value of the reward function is larger; when the value of |P| is smaller, that is, the absolute value of the above sum is smaller, the value of the reward function is larger; when the above image clarity reaches the preset image clarity threshold, the value of the reward function is larger. In other words, the above α·C can be regarded as a reward for image clarity, that is, a positive reward based on image clarity; the above -β·|P| can be regarded as a parameter penalty term, that is, in order to avoid jitter caused by frequent adjustment of acquisition parameters, a negative reward is given to the adjustment amplitude of the acquisition parameters; the above γ·G can be regarded as a goal achievement reward, that is, when the image clarity reaches the preset image clarity threshold, an additional positive reward is given.

[0116] In some embodiments, in order to avoid the reward function giving too much punishment to actions with too large adjustment amplitude (such as adjusting focal length or aperture size), which may cause the agent (such as imitation learning model) to avoid some necessary large adjustments, thus failing to achieve the ideal clarity and only learning a suboptimal autofocus strategy, the present application designs a mechanism for dynamically adjusting weights for the reward function, so that when the agent approaches the preset clarity threshold, the punishment for actions with too large adjustment amplitude is reduced, and the weight of the positive reward brought by clarity is increased. The expression of the reward function can be:

[0117] R=α·C-β·w(C,|P|)·|P|+γ·G;

[0118] Wherein, C represents the clarity; |P| represents the sum of the absolute values ​​of the above changes; G is equal to 1 or 0, 1 represents that the image clarity reaches the preset image clarity threshold, and 0 represents that the image clarity does not reach the preset image clarity threshold; α represents the first weight coefficient; β represents the second weight coefficient; γ represents the third weight coefficient, γ>α>β>0; w(C,|P|) represents a dynamic adjustment function, and the value of w(C,|P|) decreases when C and |P| gradually increase. The dynamic adjustment function is obtained based on the image clarity and the sum of the above changes. The role of the above dynamic adjustment function w(C) is to constrain the reward function of the intelligent agent (such as the imitation learning model) to avoid the occurrence of local optimal situations, that is, to avoid the imitation learning model only learning a suboptimal autofocus strategy.

[0119] It is understandable that when designing the above reward function, the most important thing is to consider the trade-offs between different factors and ensure that the reward function can correctly guide the agent (such as the imitation learning model) to learn the desired clarity. If the imitation learning model needs to significantly adjust the above acquisition parameters to obtain the optimal image clarity, then the penalty term should be adjusted appropriately to ensure that the agent can effectively achieve the goal while improving the accuracy and response speed of focus.

[0120] In some embodiments, the expression of w(C,|P|) can be:

[0121]

[0122] Wherein, the above b and k are both preset hyperparameters. For example, when the acquisition parameters include focal length f and aperture size a, P ＝ f+a; when the acquisition parameters include focal length f, P ＝ f; When the acquisition parameters include aperture size a, P ＝ a. Those skilled in the art can determine the specific values ​​of b and k according to actual scenario requirements.

[0123] In some embodiments, step S300 further includes: the image acquisition unit performs automatic focusing according to the optimal acquisition parameters to obtain a current image frame with higher definition.

[0124] It can be seen that in some embodiments, in practical applications (such as the reasoning stage), the present autofocus method can predict the optimal acquisition parameters (such as focal length) according to the clarity characteristics of the current image frame sequence through the imitation learning model, thereby realizing automatic focusing. Compared with the traditional heuristic focusing method, the present autofocus method can accurately learn the law of clarity change of the image frame sequence at different focal lengths, has better adaptability and generalization ability, and can adapt to the video focusing requirements under different scenes and shooting conditions. This is because, on the one hand, the adaptability and generalization of the present method come from the clarity evaluation model, wherein, during the training process of the clarity evaluation model, the clarity evaluation model can automatically learn and extract the features of the current image frame, such as edges, textures and other visual information related to clarity; on the other hand, the adaptability and generalization of the present method are also related to the composition of the clarity training set, that is, various lighting conditions and complex background image frames are introduced into the clarity training set, thereby improving the robustness of the present method. In addition, with the training optimization of the imitation learning model, the imitation learning model gradually learns the law of change of the clarity of the image frame under different focal length settings.

[0125] It can be seen that in some embodiments, the method can use deep learning technology to establish a clarity assessment model, and the clarity of image frames at different focal lengths can be accurately assessed through the clarity assessment model.

[0126] It can be seen that in some embodiments, the training process of the imitation learning model of the present method adopts a modifiable imitation learning, that is, by imitating the expert operation and using a labeled data set (such as the above-mentioned imitation learning training set) for training, the clarity change law of the image frame sequence at different focal lengths is learned.

[0127] It can be seen that in some embodiments, the method can obtain the clarity of the current image frame through the clarity evaluation model, and the imitation learning model predicts the optimal acquisition parameters (such as the optimal focal length) according to the clarity characteristics of the current image frame to achieve the automatic focusing function. Compared with the traditional heuristic focusing method, the method has better adaptability and generalization ability, and can adapt to the focusing requirements under different scenes and shooting conditions.

[0128] The above is some explanation about the auto focus method. Figure 5 In some embodiments of the present application, an automatic focusing device is also disclosed. The automatic focusing device includes:

[0129] The image acquisition unit 100 is configured to acquire the current image frame;

[0130] The definition acquisition unit 200 is configured to acquire the definition of the current image frame using the trained definition evaluation model;

[0131] The optimal strategy output unit 300 is configured to determine the state space and action space of the imitation learning model; wherein the state space includes the above-mentioned clarity, and the action space includes the acquisition parameters when the image acquisition unit 100 acquires the current image frame; the trained imitation learning model predicts based on the clarity to obtain the optimal acquisition parameters; wherein the image acquisition unit 100 can complete automatic focusing according to the optimal acquisition parameters.

[0132] In some embodiments, the above acquisition parameters may include focal length.

[0133] In some embodiments, the acquisition parameters may include focal length and / or aperture size.

[0134] In some embodiments, the state space may further include the current image frame.

[0135] In some embodiments, the trained imitation learning model is obtained through the following training process:

[0136] Initial training step S210: using the current imitation learning training set to train the imitation learning model and obtain an imitation learning model that has completed initial training; wherein the current imitation learning training set includes a plurality of second image frames and first training optimal acquisition parameters manually annotated for the second image frames;

[0137] Preliminary prediction step S220: the image acquisition unit 100 acquires a third image frame based on the training optimal acquisition parameters, and obtains the training clarity of the third image frame using the trained clarity evaluation model;

[0138] Expert labeling step S230: manually labeling the third image frame with the optimal acquisition parameters for training; using each third image frame and the corresponding optimal acquisition parameters for training as a new imitation learning training set;

[0139] Step S240 of updating the data set: merging the new imitation learning training set into the current imitation learning training set to obtain a new imitation learning training set;

[0140] Step S250 of stopping iteration: repeating steps S210 of initial training to step S240 of updating the data set until the condition of stopping iterative training is reached, and the imitation learning model at this time is regarded as the imitation learning model of completed training.

[0141] It should be noted that the specific process and corresponding technical effects of focusing of the present autofocus device are basically the same as the specific process and corresponding technical effects of the above-mentioned autofocus method, so the present autofocus device will not be described in detail here.

[0142] The above is some description about the auto-focus device. In some embodiments of the present application, a computer-readable storage medium is also disclosed, including a program, and the program can be executed by a processor to implement a method as in any embodiment of the present invention.

[0143] In some embodiments of the present application, a computer program product is also disclosed, including a computer program and / or instructions, and when the computer program and / or instructions are executed by a processor, a method of any embodiment of the present invention is implemented.

[0144] This document is described with reference to various exemplary embodiments. However, those skilled in the art will recognize that changes and modifications may be made to the exemplary embodiments without departing from the scope of this document. For example, various operating steps and components for performing the operating steps may be implemented in different ways (e.g., one or more steps may be deleted, modified, or incorporated into other steps) depending on the specific application or considering any number of cost functions associated with the operation of the system.

[0145] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. In addition, as understood by those skilled in the art, the principles of this article can be reflected in a computer program product on a computer-readable storage medium, which is pre-installed with a computer-readable program code. Any tangible, non-temporary computer-readable storage medium can be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD to ROM, DVD, Blu Ray disks, etc.), flash memory and / or the like. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer or other programmable data processing device to form a machine, so that these instructions executed on a computer or other programmable data processing device can generate a device that implements a specified function. These computer program instructions can also be stored in a computer-readable memory, which can instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory can form a manufactured product, including an implementation device that implements a specified function. Computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operating steps are performed on a computer or other programmable device to generate a computer-implemented process, so that the instructions executed on a computer or other programmable device can provide steps for implementing a specified function.

[0146] Although the principles of this invention have been shown in various embodiments, many modifications of structures, arrangements, proportions, elements, materials and components particularly suitable for specific environments and operational requirements can be used without departing from the principles and scope of this invention. The above modifications and other changes or amendments will be included in the scope of this invention.

[0147] The foregoing specific description has been described with reference to various embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present disclosure. Therefore, the consideration of the present disclosure will be illustrative rather than restrictive, and all these modifications will be included in its scope. Similarly, the advantages, other advantages and solutions to the problems of various embodiments have been described above. However, the benefits, advantages, solutions to the problems and any elements that can produce these, or make them more clear, should not be interpreted as critical, necessary or necessary. The term "include" and any other variants used in this article are all non-exclusive inclusions, so that the process, method, article or device including the list of elements not only includes these elements, but also includes other elements that are not explicitly listed or do not belong to the process, method, system, article or device. In addition, the term "coupled" and any other variants used in this article refer to physical connections, electrical connections, magnetic connections, optical connections, communication connections, functional connections and / or any other connections.

[0148] Those skilled in the art will appreciate that many changes may be made to the details of the above-described embodiments without departing from the basic principles of the invention. Therefore, the scope of the present invention should be determined solely by the claims.

Claims

1. An automatic focusing method, characterized in that: include: The image acquisition unit acquires the current image frame; and obtains the clarity of the current image frame using the trained clarity assessment model; Determine the state space and action space of the imitation learning model; wherein the state space includes the clarity, the action space includes the acquisition parameters used by the image acquisition unit when acquiring the current image frame, and the acquisition parameters include focal length and / or aperture size; The imitation learning model predicts the clarity to obtain the optimal acquisition parameters, so as to use the optimal acquisition parameters to acquire the next image frame.

2. The method according to claim 1, characterized in that The trained clarity assessment model is obtained through the following training process: Constructing an initial clarity assessment model; Acquire a clarity training set; wherein the clarity training set includes a plurality of first image frames and annotations of the first image frames; the annotations are the clarity of the first image frames; The initial clarity assessment model is trained using the clarity training set to obtain a trained clarity assessment model.

3. The method according to claim 2, characterized in that The step of obtaining a clarity training set comprises: The image acquisition unit acquires images of the first training scene to obtain the plurality of first image frames; Calculating the clarity of each of the first image frames, and using the clarity as a label of the first image frame; wherein the acquisition parameters and / or the clarity corresponding to each of the first image frames are different; Each of the first image frames and the annotations of the first image frames are used as the clarity training set.

4. The method according to claim 3, characterized in that The light conditions when collecting each of the first image frames are different; the first training scene contains moving objects and / or objects that are changing.

5. The method according to claim 1, characterized in that The imitation learning model is obtained through the following training process: The steps of initial training are: performing image acquisition on a second training scene to obtain a plurality of second image frames; using the current imitation learning training set to train the imitation learning model and obtain an imitation learning model that has completed initial training; wherein the imitation learning training set includes the first clarity of the second image frame and the training optimal acquisition parameters manually annotated for the second image frame; the training optimal acquisition parameters include focal length; the first clarity is obtained by the clarity evaluation model; The step of preliminary prediction: performing image acquisition on the third training scene to obtain a plurality of third image frames, and obtaining a first definition of the third image frames; the imitation learning model that has completed the initial training performs prediction according to the first definition to obtain a first training optimal acquisition parameter; Expert labeling step: manually labeling the third image frame with the second optimal acquisition parameter for training; using the first definition and the second optimal acquisition parameter for training of the third image frame as a new imitation learning training set; The step of updating the data set is as follows: obtaining a new imitation learning training set according to the new imitation learning training set; The step of stopping iteration: repeating the steps of the initial training to the step of updating the data set until the condition of stopping iterative training is reached, and using the imitation learning model at this time as the imitation learning model of the completed training.

6. The method according to claim 5, characterized in that The initial training step also includes: determining a reward function of the imitation learning model; The preliminary prediction step also includes: the image acquisition unit performs automatic focusing according to the first training optimal acquisition parameter, and again acquires the image of the third training scene to obtain a new third image frame; and acquires a second definition of the new third image frame; The reward function is obtained based on the difference between the second clarity and the first clarity; and the condition for stopping iterative training is: the value of the reward function reaches a preset reward value threshold.

7. An automatic focusing device, characterized in that: include: An image acquisition unit, configured to acquire a current image frame; A definition acquisition unit is configured to acquire the definition of the current image frame using the trained definition evaluation model; The optimal strategy output unit is configured to determine the state space and action space of the imitation learning model; wherein the state space includes the clarity, and the action space includes the acquisition parameters of the image acquisition unit when acquiring the current image frame, and the acquisition parameters include focal length and / or aperture size; the trained imitation learning model predicts based on the clarity to obtain the optimal acquisition parameters; wherein the image acquisition unit can complete automatic focusing based on the optimal acquisition parameters.

8. The automatic focusing device according to claim 7, wherein: The imitation learning model that has completed the training is obtained through the following training process: The steps of initial training: the image acquisition unit acquires images of the second training scene to obtain a plurality of second image frames; the imitation learning model is trained using the current imitation learning training set to obtain an imitation learning model that has completed initial training; wherein the imitation learning training set includes the first clarity of the second image frame and the training optimal acquisition parameters manually annotated for the second image frame; the training optimal acquisition parameters include focal length; the first clarity is obtained by the clarity evaluation model; The step of preliminary prediction: the image acquisition unit acquires images of the third training scene to obtain a plurality of third image frames, and obtains a first definition of the third image frames; the imitation learning model that has completed the initial training performs prediction according to the first definition to obtain a first training optimal acquisition parameter; Expert labeling step: manually labeling the third image frame with the second optimal acquisition parameter for training; using the first definition and the second optimal acquisition parameter for training of the third image frame as a new imitation learning training set; The step of updating the data set is as follows: obtaining a new imitation learning training set according to the new imitation learning training set; The step of stopping iteration: repeating the steps of the initial training to the step of updating the data set until the condition of stopping iterative training is reached, and using the imitation learning model at this time as the imitation learning model of the completed training.

9. A computer-readable storage medium, characterized in that: The method comprises a program which can be executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images