Automatic focusing method, device, electronic device and computer readable storage medium

Abstract

The embodiments of the present application disclose an autofocus method, device, electronic device, and computer-readable storage medium. The method includes: determining the first image area information corresponding to the target object in the current frame image; predicting the first predicted area information of the target object in the Nth frame image after the current frame image based on the first image area information, where N is a positive integer; using the first predicted area information as an observation value, processing the first image area information and the observation value through a filter, and predicting the second predicted area information of the target object in the Nth frame image after the current frame image; and controlling the focus module to perform a focus operation based on the second predicted area information. The above-mentioned autofocus method, device, electronic device, and computer-readable storage medium can improve the accuracy of autofocus and improve the imaging effect of the target object.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to an autofocus method, device, electronic device and computer-readable storage medium. Background Art

[0002] When the camera is capturing images, it needs to focus first to ensure that the target object is clearly imaged in the image. For a target object in motion, the camera needs to track the target object in real time and determine the actual object distance of the target object, so as to move the lens to the focal plane where the target is clear and ensure that the moving target is clearly focused.

[0003] However, in actual use of the camera device, in order to ensure the imaging effect after the image of the target object is captured, the captured image usually needs to be fully processed first, including autofocus, auto exposure, white balance processing and noise reduction, etc. Therefore, the tracking process of the target object will be delayed, and the tracking result of the target object by the camera device will deviate from the actual position of the target object, resulting in inaccurate autofocus, which reduces the imaging effect of the target object. Summary of the invention

[0004] The embodiments of the present application disclose an autofocus method, device, electronic device and computer-readable storage medium, which can improve the accuracy of autofocus and improve the imaging effect of the target object.

[0005] The present application discloses an automatic focusing method, including:

[0006] Determine first image region information corresponding to the target object in the current frame image;

[0007] Predicting first predicted region information of the target object in an N-th frame image after the current frame image according to the first image region information, where N is a positive integer;

[0008] The first predicted region information is used as an observation value, and the first image region information and the observation value are processed by a filter to predict and obtain second predicted region information of the target object in the Nth frame image after the current frame image;

[0009] The focusing module is controlled to perform a focusing operation based on the second prediction area information.

[0010] The embodiment of the present application discloses an automatic focusing device, comprising:

[0011] An information determination module, used to determine first image region information corresponding to the target object in the current frame image;

[0012] A first prediction module, configured to predict first prediction area information of the target object in an N-th frame image after the current frame image according to the first image area information, where N is a positive integer;

[0013] A second prediction module is used to use the first prediction area information as an observation value, process the first image area information and the observation value through a filter, and predict the second prediction area information of the target object in the Nth frame image after the current frame image;

[0014] A control module is used to control the focus module to perform a focus operation based on the second prediction area information.

[0015] An embodiment of the present application discloses an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method described above.

[0016] An embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is implemented.

[0017] The autofocus method, device, electronic device and computer-readable storage medium disclosed in the embodiments of the present application determine the first image area information corresponding to the target object in the current frame image, predict the first predicted area information of the target object in the Nth frame image after the current frame image based on the first image area information, use the first predicted area information as an observation value, process the first image area information and the observation value through a filter, predict the second predicted area information of the target object in the Nth frame image after the current frame image, and control the focus module to perform a focus operation based on the second predicted area information. By predicting the focus area of ​​the target object in the Nth frame image after the current frame image, and then focusing based on the focus area, the deviation between the tracking result and the actual position of the target object can be reduced, thereby improving the accuracy of the autofocus. Moreover, the first predicted area information is used as the observation value of the filter for prediction, which improves the accuracy of the prediction result and the imaging effect of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1A A block diagram of an electronic device implementing target tracking in an ideal state in an embodiment;

[0020] Figure 1B A block diagram of an electronic device implementing target tracking in actual use in an embodiment;

[0021] Figure 2 is a block diagram of an image processing circuit in one embodiment;

[0022] Figure 3 is a flow chart of an automatic focusing method in one embodiment;

[0023] Figure 4 is a flow chart of an automatic focusing method in another embodiment;

[0024] Figure 5A A schematic diagram of determining a target subject in one embodiment;

[0025] Figure 5B A schematic diagram of determining a target subject in another embodiment;

[0026] Figure 6 A flowchart of obtaining first prediction area information in one embodiment;

[0027] Figure 7 A flowchart of obtaining second prediction area information in one embodiment;

[0028] Figure 8 It is a schematic diagram of using a filter to predict the Nth frame image of the target object after the current frame image in one embodiment;

[0029] Fig. 9 is a block diagram of an automatic focusing device in one embodiment;

[0030] Fig.10 FIG. 4 is a structural block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0033] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, the first prediction area information may be referred to as the second prediction area information, and similarly, the second prediction area information may be referred to as the first prediction area information. Both the first prediction area information and the second prediction area information are predicted image area information, but they are not the same predicted image area information.

[0034] Figure 1A FIG. 1 is a block diagram of an electronic device implementing target tracking in an ideal state in an embodiment. Figure 1A As shown, the electronic device may include an image sensor 110, a target determination module 120, a tracking module 130 and a focusing module 140. Focus tracking refers to tracking and focusing on a moving target. The target determination module 120 can be used to determine the target object to be tracked. During the movement of the target object, the image sensor 110 can capture image data. The tracking module 130 can determine the position of the target object in each frame of the image based on the image data captured by the image sensor 110, and track the target object. The focusing module 140 can focus based on the tracking result of the tracking module 130, and can use the position of the target object in the image output by the tracking module 130 as the focus area to determine the actual object distance of the target object, thereby moving the lens to a focal plane where the target is clear to ensure that the target object is clearly focused.

[0035] In an ideal focus tracking process, the focus module 140 is placed after the tracking module 130, so that the tracking result received by the focus module 140 each time is the position of the target object in the current frame image, thereby ensuring accurate focus. However, in the actual focus tracking process, in order to ensure the imaging effect, it is usually required that the image data captured by the image sensor 110 is fully processed, and the processing may include but is not limited to auto focus, auto exposure, white balance processing and noise reduction, etc. Therefore, the tracking module 130 is placed after the focus module 140.

[0036] Figure 1B FIG. 1 is a block diagram of an electronic device implementing target tracking in actual use in an embodiment. Figure 1BAs shown, the tracking module 130 is placed after the focusing module 140, and the focusing module 140 is at the front end position during the image acquisition process, which will cause the tracking result output by the tracking module 130 to be delayed in being fed back to the focusing module 140, that is, the tracking result output by the tracking module 130 is the tracking result of the previous frames of images, rather than the tracking result of the target object in the image that the focusing module 140 currently needs to focus on. For a target object in motion, the delayed feedback of the tracking module 130 will cause a deviation between the tracking result of the target object and the actual position of the target object, resulting in an inaccurate focus area of ​​the focusing module 140, and the target object will not be clear in the acquired image, which reduces the imaging effect of the target object.

[0037] The embodiments of the present application provide an autofocus method, device, electronic device and computer-readable storage medium, which can predict the focus area of ​​the target object in the Nth frame image after the current frame image, and then focus based on the predicted focus area, thereby reducing the deviation between the tracking result and the actual position of the target object, and improving the accuracy of autofocus.

[0038] An embodiment of the present application provides an electronic device. The electronic device includes an image processing circuit, which can be implemented using hardware and / or software components and can include various processing units that define an ISP (Image Signal Processing) pipeline. Figure 2 FIG. 1 is a block diagram of an image processing circuit in one embodiment. For ease of explanation, Figure 2 Only various aspects of the image processing technology related to the embodiments of the present application are shown.

[0039] like Figure 2As shown, the image processing circuit includes an ISP processor 240 and a control logic 250. The image data captured by the imaging device 210 is first processed by the ISP processor 240, and the ISP processor 240 analyzes the image data to capture image statistics that can be used to determine one or more control parameters of the imaging device 210. The imaging device 210 may include one or more lenses 212 and an image sensor 214. The image sensor 214 may include a color filter array (such as a Bayer filter), and the image sensor 214 may obtain light intensity and wavelength information captured by each imaging pixel, and provide a set of raw image data that can be processed by the ISP processor 240. The attitude sensor 220 (such as a three-axis gyroscope, a Hall sensor, an accelerometer, etc.) can provide the acquired image processing parameters (such as anti-shake parameters) to the ISP processor 240 based on the interface type of the attitude sensor 220. The attitude sensor 220 interface can use an SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

[0040] It should be noted that although Figure 2 Only one imaging device 210 is shown in the figure, but in the embodiment of the present application, at least two imaging devices 210 may be included, each imaging device 210 may correspond to one image sensor 214, or multiple imaging devices 210 may correspond to one image sensor 214, which is not limited here. The working process of each imaging device 210 may refer to the above description.

[0041] In addition, the image sensor 214 may also send raw image data to the gesture sensor 220 , and the gesture sensor 220 may provide the raw image data to the ISP processor 240 based on the interface type of the gesture sensor 220 , or the gesture sensor 220 may store the raw image data in the image memory 230 .

[0042] The ISP processor 240 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 240 may perform one or more image processing operations on the raw image data and collect statistical information about the image data. The image processing operations may be performed at the same or different bit depth precisions.

[0043] The ISP processor 240 may also receive image data from the image memory 230. For example, the gesture sensor 220 interface sends raw image data to the image memory 230, and the raw image data in the image memory 230 is then provided to the ISP processor 240 for processing. The image memory 230 may be a part of a memory device, a storage device, or an independent dedicated memory within an electronic device, and may include a DMA (Direct Memory Access) feature.

[0044] When receiving raw image data from the image sensor 214 interface or from the posture sensor 220 interface or from the image memory 230, the ISP processor 240 may perform one or more image processing operations, such as time domain filtering. The processed image data may be sent to the image memory 230 for further processing before being displayed. The ISP processor 240 receives the processed data from the image memory 230 and performs image data processing in the original domain and in the RGB and YCbCr color spaces on the processed data. The image data processed by the ISP processor 240 may be output to the display 260 for viewing by the user and / or further processed by a graphics engine or GPU (Graphics Processing Unit). In addition, the output of the ISP processor 240 may also be sent to the image memory 230, and the display 260 may read the image data from the image memory 230. In one embodiment, the image memory 230 may be configured to implement one or more frame buffers.

[0045] The statistical data determined by the ISP processor 240 may be sent to the control logic 250. For example, the statistical data may include image sensor 214 statistical information such as vibration frequency of a gyroscope, automatic exposure, automatic white balance, automatic focus, flicker detection, black level compensation, lens 212 shading correction, etc. The control logic 250 may include a processor and / or microcontroller that executes one or more routines (such as firmware), and the one or more routines may determine control parameters of the imaging device 210 and control parameters of the ISP processor 240 based on the received statistical data. For example, the control parameters of the imaging device 210 may include attitude sensor 220 control parameters (such as gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, camera anti-shake displacement parameters, lens 212 control parameters (such as focal length for focus or zoom), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (for example, during RGB processing), and lens 212 shading correction parameters.

[0046] For example, in combination Figure 2The image processing circuit of the present application is used to illustrate the autofocus method provided in the embodiment of the present application. After the image sensor 214 in the image processing circuit captures the original image data of the first frame, it is transmitted to the ISP processor 240. The ISP processor 240 can determine the target object contained in the original image data, and obtain information such as the initial position of the target object in the first frame image and the size of the initial area occupied. The ISP processor 240 can send the information such as the initial position and the size of the initial area occupied to the control logic 250, and the control logic 250 generates corresponding lens control parameters to control the movement of the lens of the imaging device 210 to achieve focusing on the target.

[0047] After the imaging device 210 has completed focusing, the image sensor 214 continues to capture raw image data and sends it to the ISP processor 240. The ISP processor 240 can process the raw image data to obtain the current frame image. The ISP processor 240 can determine the first image area information corresponding to the target object in the current frame image, and predict the first predicted area information of the target object in the Nth frame image after the current frame image based on the first image area information, and then use the first predicted area information as an observation value, process the first image area information and the observation value through a filter, and predict the second predicted area information of the target object in the Nth frame image after the current frame image. The ISP processor 240 can send the second predicted area information to the control logic 250, and the control logic 250 determines the focus area based on the second predicted area information, and generates corresponding lens control parameters to control the lens movement, so that the imaging device 210 focuses on the focus area, thereby achieving accurate tracking of the target device.

[0048] like Figure 3 As shown, in one embodiment, an image processing method is provided, which can be applied to the above-mentioned electronic devices, which may include but are not limited to mobile phones, tablet computers, smart wearable devices, vehicle-mounted terminals, monitoring systems, laptop computers and PCs (Personal Computers), etc. The electronic device may be provided with one or more cameras, or may be connected to one or more external camera devices. The image processing method may include the following steps:

[0049] Step 310: Determine first image region information corresponding to the target object in the current frame image.

[0050] The target object may refer to the object of focus of the electronic device, that is, the main shooting object of the camera. The target object may include but is not limited to various objects such as people, animals, inanimate objects, etc. The target object may be the entire object or only a part of the object. For example, the target object may be a whole person or a person's face. The target object may be an entire building or a door or floor of a building.

[0051] The electronic device can acquire an image containing a target object captured by a camera in real time, and process each frame of the image acquired most recently to obtain first image region information corresponding to the target object in each frame of the image. The current frame image can refer to an image obtained by the processor of the electronic device processing the raw image data captured by the image sensor of the camera most recently.

[0052] The electronic device may identify the current frame image, determine the image area occupied by the target object in the current frame image, and obtain first image area information corresponding to the image area. Optionally, the first image area information may include at least one of position information and size information, wherein the position information may refer to the position coordinates of the target object in the current frame image, and the size information may refer to the size of the image area occupied by the target object in the current frame image.

[0053] Step 320 : predicting first predicted region information of the target object in the Nth frame image after the current frame image according to the first image region information.

[0054] The electronic device may predict the image area occupied by the target object in the Nth frame image after the current frame image based on the first image area information of the target object in the current frame image to obtain first predicted area information. The first predicted area information may include at least one of first predicted position information and first predicted size information.

[0055] N may be a positive integer. In some embodiments, N may be set according to actual needs, such as 1, 2, 3, etc. In some embodiments, N may be used to characterize the number of delayed frames, which may refer to the number of frames of delayed feedback from the tracking module 130 to the focusing module 140, that is, the difference between the number of frames corresponding to the tracking result fed back by the tracking module 130 and the number of frames focused by the focusing module 140 based on the tracking result. For example, when the tracking module 130 identifies the second frame of the image and obtains the tracking result of the target object, and then feeds back to the focusing module 140, the focusing module 140 focuses on the fifth frame of the image based on the tracking result, then there are 3 frames of delayed frames between the tracking module 130 and the focusing module 140.

[0056] Optionally, the above-mentioned delay frame number may be related to the performance of the processor. The faster the processing speed of the processor, the shorter the delay frame number may be, and the slower the processing speed of the processor, the longer the delay frame number may be. The delay frame number may be the frame number obtained by the electronic device through a large number of tests before leaving the factory, or it may be obtained by the electronic device through field testing each time it tracks a new target object, or the electronic device may retest the delay frame number at fixed intervals.

[0057] Taking the field test performed by the electronic device each time it tracks a new target object as an example, each time the electronic device needs to track a new target object, it can determine the target object to be tracked through the initial frame image, and determine the area information of the target object in the initial frame image. The electronic device can first control the focus module to focus on the area information of the target object in the initial frame image as the focus area, and collect a new image, and then perform area prediction based on the new image to obtain the first predicted focus area, and control the focus module to focus on the first predicted focus area. The number of frames of the image obtained by the focus module focusing on the area information of the target object in the initial frame image as the focus area can be recorded, and the number of frames of the image obtained by the focus module focusing on the first predicted focus area can be recorded, and the difference between the two recorded frame numbers can be calculated, and the difference is the delayed frame number.

[0058] Step 330 , taking the first predicted region information as the observation value, processing the first image region information and the observation value through a filter, and predicting the second predicted region information of the target object in the Nth frame image after the current frame image.

[0059] In an embodiment of the present application, the filter may include a Kalman filter, which is a data filtering model that uses the state of a linear system and can estimate the state of a dynamic system from a series of data with measurement noise based on known situations. The Kalman filter can estimate the state of the target object in the Nth frame image after the current frame image based on the first image area information, and can use the first predicted area information as the observation value of the Kalman filter, and use the observation value to correct the state estimated by the Kalman filter, thereby obtaining the second predicted area information of the target object in the Nth frame image after the current frame image. At the same time, using the real data of the target object in the current frame image (such as the first image area) and the first predicted area information of the Nth frame image after the current frame image, based on the filter, prediction can obtain more accurate prediction results.

[0060] In the related technology, the Kalman filter needs to measure the state at the target moment and modify the estimated state with the real value obtained by the real measurement as the observation value. In the present application, the first prediction area information is directly used as the observation value, which reduces the data processing time and improves the prediction efficiency while ensuring the accuracy of the prediction result.

[0061] Step 340: Control the focus module to perform a focus operation based on the second prediction area information.

[0062] The second predicted area information may include at least one of second predicted position information and second predicted size information. The focus module may determine the focus area based on the second predicted area information, may use the second predicted position information as the position of the focus area, use the second predicted size information as the size of the focus area, and perform a focus operation on the focus area.

[0063] In some embodiments, the focus module may perform the focus operation using a focus method such as phase focus or contrast focus. In the phase focus method, the lens position of the camera may be adjusted according to the phase difference information of each pixel point contained in the focus area in the original image data captured by the image sensor, so that the phase difference of each pixel point contained in the focus area is 0, which indicates that the focus area is a clear image area, thereby achieving the focus on the focus area. In the contrast focus method, the clarity of the focus area in the original image data captured by the image sensor may be calculated, and the lens position of the camera may be adjusted based on the clarity. A hill climbing algorithm may be used to find the best lens position, and the focus area corresponding to the best lens position has the highest clarity, thereby achieving the focus on the focus area. It should be noted that the focus module may also perform the focus operation in other ways, which are not limited here.

[0064] In an embodiment of the present application, the first image area information corresponding to the target object in the current frame image is determined, and the first predicted area information of the target object in the Nth frame image after the current frame image is predicted based on the first image area information. The first predicted area information is used as an observation value, and the first image area information and the observation value are processed by a filter to predict the second predicted area information of the target object in the Nth frame image after the current frame image, and the focus module is controlled to perform a focus operation based on the second predicted area information. By predicting the focus area of ​​the target object in the Nth frame image after the current frame image, and then focusing based on the focus area, the deviation between the tracking result and the actual position of the target object can be reduced, and the accuracy of the autofocus is improved. Moreover, the first predicted area information is used as the observation value of the filter for prediction, which improves the accuracy of the prediction result and the imaging effect of the target object.

[0065] like Figure 4 As shown, in one embodiment, another automatic focusing method is provided, which can be applied to the above electronic device. The automatic focusing method may include the following steps:

[0066] Step 402: determine the target object from the initial frame image, and obtain the initial image area of ​​the target object in the initial frame image.

[0067] The initial frame image may refer to the image of the target object to be tracked that is first acquired by the electronic device. The electronic device may determine the image position of the target object in the initial frame image, and acquire the initial image area of ​​the target object in the initial frame image based on the image position. Optionally, the image area occupied by the target object in the initial frame image may be identified, and the area of ​​the circumscribed quadrilateral (such as a rectangle, square, etc.) of the occupied image area may be used as the initial image area. Alternatively, the image position of the target object in the initial frame image may be taken as the center, and the initial image area may be selected according to a set image area size.

[0068] In some embodiments, the electronic device may determine the target object from the initial frame image in the following ways, including but not limited to:

[0069] Method 1: Use a pre-trained subject recognition model to perform subject recognition on images captured by the camera to determine the target object. When the subject recognition model recognizes a new target object, the image of the new target object can be used as the initial frame image, and the electronic device can control the camera to focus on the new target object.

[0070] The subject recognition model can be trained based on a large number of sample images with subject labels. In each training image, the image position of the subject and the size of the image area occupied can be labeled. Furthermore, the category of the subject can be annotated. For example, if the subject in the sample image is a face, it can be labeled as a face image; if the subject in the sample image is a scene, it can be labeled as a scene image; if the subject in the sample image is food, it can be labeled as a food image, etc., but not limited to this. The training image can be input into the neural network for training, and the parameters of the neural network are continuously adjusted according to the recognition results output by the neural network and the annotation information carried by the sample image, so that the error between the recognition result output by the neural network and the annotation information is less than the error threshold, thereby obtaining the subject recognition model.

[0071] Figure 5A FIG. 1 is a schematic diagram of determining a target subject in an embodiment. Figure 5A As shown, Figure 5A The image in is input into the subject recognition model, and the subject recognition model can recognize that the target object is a human face, and then the initial image area 510 can be determined according to the circumscribed quadrilateral of the area occupied by the human face in the image. The subject recognition model obtained by machine learning is used to determine the target object, which can improve the efficiency and accuracy of recognition.

[0072] Method 2: The electronic device may display the image captured by the camera, and when a target selection operation performed by the user on the displayed image is detected, the target object may be determined according to the target selection operation, and the image in which the target selection operation is detected may be used as the initial frame image. Optionally, the target selection operation may include a touch operation, a voice operation, and a sight interaction operation, but is not limited thereto.

[0073] Taking a touch operation as an example, the electronic device can obtain the touch coordinates of the touch operation on the screen and convert the touch coordinates into image coordinates in the displayed image, so that the target object can be determined according to the image coordinates. The image coordinates obtained by the conversion can be used as the center, and the image area where the target object is located can be selected according to the set image area size. The image area where the target object is located is the initial image area. Figure 5B FIG. 4 is a schematic diagram of determining a target subject in another embodiment. Figure 5B As shown, the electronic device can Figure 5B If the user touches the position of the flower in the image, the electronic device can use the image coordinates obtained by converting the touch coordinates as the center and select the initial image area 520 according to the set image area size.

[0074] Taking voice operation as an example, the electronic device can receive the user's voice information, recognize the voice information, and obtain the voice content. If the voice content includes subject information, the target object can be searched and determined in the displayed image according to the subject information. The subject information may include but is not limited to the category name of the target object, the position of the target object in the displayed image, etc. For example, if the voice information includes "face", the face can be searched in the displayed image and the face can be determined as the target object. If the voice information includes "animal", the animal can be searched in the displayed image and the searched animal can be determined as the target object, etc., but it is not limited to this.

[0075] Taking the line of sight interaction operation as an example, after displaying an image, the electronic device can collect the user's eye image through the eye tracking system, extract the eye features in the eye image, analyze the eye features to obtain the user's eye focus on the screen, and determine the target object based on the line of sight focus. The image area where the target object is located can be selected according to the set image area size with the line of sight focus as the center, and the image area where the target object is located is the initial image area. Determining the target object based on the user's target selection operation can make the determined target object more in line with user needs and improve user stickiness.

[0076] It should be noted that other methods may also be used to determine the target object in the initial frame image, and are not limited to the above methods.

[0077] Step 404 , matching the current frame image with the initial image region, and determining the target region in the current frame image that has the highest similarity with the initial image region.

[0078] The electronic device may track the target object based on the initial image area of ​​the target object in the initial frame image, may align the current frame image with the initial image area, and determine the target area in the current frame image that matches the initial image area. In some embodiments, image features of the initial image area may be extracted, and when matching the current frame image with the initial image area, the image features of the current frame image may be extracted, the image features of the initial image area may be compared with the image features of the current frame image, and the image features in the current frame image that are most similar to the image features of the initial image area are used as matching image features, and the target area is determined according to the image area in the current frame image that matches the initial image area.

[0079] In some embodiments, the current frame image and the initial image area can also be matched by correlation filtering. The current frame image and the initial image area can be converted to the frequency domain respectively, and the correlation between the current frame image and the initial image area is calculated in the frequency domain. The area with the highest correlation in the current frame image is the target area with the highest similarity to the initial image area. Furthermore, the frequency domain dot multiplication can be performed on the current frame image and the initial image area in the frequency domain to obtain the correlation between different image areas of the current frame image and the initial image area. By replacing the convolution result in the image domain with the frequency domain dot multiplication, the amount of image calculation can be reduced and the data processing efficiency can be improved.

[0080] In other embodiments, a twin neural network model may be used to match the current frame image with the initial image area. The current frame image and the initial image area may be input into two neural network models with the same parameters, respectively, wherein one neural network model may output a first result corresponding to each image area based on the image features of different image areas of the current frame image, and the other neural network model may output a second result based on the initial image area, and compare the first result with the second result. If the difference between the first result and the second result is smaller, it means that the images are more similar, and the image area corresponding to the first result with the smallest difference from the second result may be selected as the target area in the current frame image with the highest similarity to the initial image area.

[0081] It should be noted that the method of determining the target area is not limited to the above methods, and other methods can also be used for determination, which is not limited in this application.

[0082] Step 406: Acquire first image region information corresponding to the target object in the current frame image according to the target region.

[0083] After the electronic device determines the target area in the current frame image that is most similar to the initial image area, it can use the position coordinates of the target area as the position information of the target object in the current frame image. The position coordinates of the target area can be the center coordinates of the target area, or the coordinates of a corner point selected in the target area (for example, the lower left corner coordinates, the upper right corner coordinates, etc.). The area size of the target area can be used as the size information of the target object in the current frame image, thereby obtaining the first image area information.

[0084] In an embodiment of the present application, the first image area information of the target object in the current frame image is determined by the initial image area of ​​the target object in the initial frame image. It is only necessary to determine the target object in the initial frame image, and there is no need to perform subject recognition in each frame image, which can improve image processing efficiency.

[0085] Step 408 : predicting first predicted region information of the target object in the Nth frame image after the current frame image according to the first image region information.

[0086] In one embodiment, Figure 6 As shown, the step of predicting first predicted region information of the target object in the Nth frame image after the current frame image according to the first image region information may include steps 602 to 604.

[0087] Step 602: Calculate the change speed information corresponding to the target object in the current frame image according to the second image region information corresponding to the target object in the historical frame image and the first image region information.

[0088] The historical frame image is an image captured by the camera before the current frame image. The historical frame image may be the previous frame image, the second previous frame image, the third previous frame image, etc. of the current frame image. For example, if the current frame image is the 8th frame image, the 7th frame image, the 6th frame image, the 5th frame image, etc. may be used as the historical frame image. The frame number difference between the historical frame image and the current frame image may be set according to actual needs. Optionally, the electronic device may calculate the corresponding change speed information of the target object in the current frame image in combination with one or more historical frame images.

[0089] In some embodiments, the frame number difference between the historical frame image and the current frame image is positively correlated with the delay time, and the delay time can be used to characterize the delay time of feedback from the tracking module 130 to the focusing module 140 .

[0090] As an implementation method, the delay time can be determined based on the duration of the first prediction area information and the second prediction area information in the Nth frame image after the current frame image of the predicted target object. The electronic device can be provided with a timer to record the starting time corresponding to the first prediction area and the feedback time of feeding back the second prediction area information to the focus module. The time difference between the feedback time and the starting time is the duration of predicting the first prediction area information and the second prediction area information, and the time difference can be used as the delay time.

[0091] As another implementation, the delay time may correspond to the aforementioned delay frame number, and the delay time may be determined according to the frame rate of the camera and the delay frame number. Specifically, the acquisition time of each frame of the camera may be calculated according to the frame rate of the camera, and the product of the acquisition time and the delay frame number may be used as the delay time. Optionally, the delay frame number may also be determined according to the duration of the first prediction area information and the second prediction area information of the predicted target object in the Nth frame image after the current frame image.

[0092] Further, the frame number difference between the historical frame image and the current frame image may be proportional to the aforementioned delayed frame number N, and the frame number difference between the historical frame image and the current frame image is an integer multiple of N. The electronic device may first obtain the delayed frame number N, and then obtain the historical frame image according to the delayed frame number N. For example, if the current frame image is the i-th frame image, the historical frame image may be the iN frame image, the i-2N frame image, etc., but is not limited thereto.

[0093] The second image area information may include at least one of the position information and size information of the target object in the historical frame image. The change speed information may include information such as the change speed and the change acceleration, wherein the change speed may include at least one of the position change speed and the size change speed, and the change acceleration may include at least one of the position change acceleration and the size change acceleration.

[0094] Step 604 : predicting first predicted region information of the target object in the Nth frame image after the current frame image according to the change speed information.

[0095] In some embodiments, the position change speed of the target object in the current frame image can be determined based on the position information of the target object in the current frame image and the position information in the historical frame image. The position change acceleration of the target object in the current frame image can be determined based on the position change speed of the target object in the current frame image and the position change speed in the historical frame image.

[0096] Furthermore, the number of delayed frames can be used as the unit time, and the position change speed of the target object in the current frame image can be the difference between the position information of the target object in the current frame image and the position information in the historical frame image. The position change speed of the target object in the current frame image can be the difference between the position change speed of the target object in the current frame image and the position change speed in the historical frame image. The position change speed and position change acceleration of the target object in the current frame image can be calculated as shown in formulas (1) to (4):

[0097]

[0098]

[0099]

[0100]

[0101] Among them, x i Indicates the position coordinate of the target object in the X-axis direction in the i-th frame image; x i-N Indicates the position coordinate of the target object in the X-axis direction in the iNth frame image; Indicates the position change speed of the target object in the X-axis direction in the i-th frame image; i Indicates the position coordinate of the target object in the Y-axis direction in the i-th frame image; y i-N Indicates the position coordinate of the target object in the Y-axis direction in the iNth frame image; Indicates the position change speed of the target object in the Y-axis direction in the i-th frame image; Indicates the position change speed of the target object in the X-axis direction in the iNth frame image; Indicates the acceleration of the position change speed of the target object in the X-axis direction in the i-th frame image; Indicates the position change speed of the target object in the Y-axis direction in the iNth frame image; Indicates the acceleration of the position change of the target object in the Y-axis direction in the i-th frame image. It should be noted that the X-axis direction and the Y-axis direction can be represented by the image coordinate system or the pixel coordinate system of the i-th frame image.

[0102] As a specific implementation, the first predicted position coordinates of the target object in the X-axis direction in the N-th frame image after the current frame image can be predicted based on the position change speed and position change acceleration of the target object in the X-axis direction in the current frame image; the first predicted position coordinates of the target object in the Y-axis direction in the current frame image can be predicted based on the position change speed and position change acceleration of the target object in the Y-axis direction in the N-th frame image after the current frame image. The position information of the target object in the current frame image can be determined as shown in formulas (5) to (6):

[0103]

[0104]

[0105] in, represents the first predicted position coordinate of the target object in the X-axis direction in the N-th frame image (i.e., the i+N-th frame image) after the i-th frame image, Indicates the first predicted position coordinates of the target object in the Y-axis direction in the i+Nth frame image. The first predicted position information may include

[0106] In some embodiments, the size change speed of the target object in the current frame image can be determined based on the size information of the target object in the current frame image and the size information in the historical frame images. The size change acceleration of the target object in the current frame image can be determined based on the size change speed of the target object in the current frame image and the size change speed in the historical frame images.

[0107] Furthermore, the number of delayed frames can be used as the unit time, and the size change speed of the target object in the current frame image can be the difference between the size information of the target object in the current frame image and the size information in the historical frame image. The size change speed of the target object in the current frame image can be the difference between the size change speed of the target object in the current frame image and the size change speed in the historical frame image. The size information may include the height and width of the image area corresponding to the target object in the image. The size change speed and size change acceleration of the target object in the current frame image can be calculated as shown in formulas (7) to (10):

[0108]

[0109]

[0110]

[0111]

[0112] Among them, w i Indicates the width of the image area corresponding to the target object in the i-th frame image; w i-N Indicates the width of the image area corresponding to the target object in the iNth frame image; h represents the wide change speed of the corresponding image area of ​​the target object in the i-th frame image; i Indicates the height of the corresponding image area of ​​the target object in the i-th frame image; h i-N Indicates the height of the corresponding image area of ​​the target object in the iNth frame image; Indicates the high change speed of the corresponding image area of ​​the target object in the i-th frame image; Indicates the wide change speed of the corresponding image area of ​​the target object in the iNth frame image; Indicates the wide variation acceleration of the target object in the corresponding image area in the i-th frame image; Indicates the high change speed of the corresponding image area of ​​the target object in the iNth frame image; Indicates the height change acceleration of the corresponding image area of ​​the target object in the i-th frame image. Optionally, the width of the image area can be the side length of the image area parallel to the X-axis direction of the image coordinate system, and the height of the image area can be the side length of the image area parallel to the Y-axis direction of the image coordinate system.

[0113] As a specific implementation, the first predicted width value of the image area corresponding to the target object in the Nth frame image after the current frame image can be predicted based on the wide change speed and wide change acceleration of the image area corresponding to the target object in the current frame image; the first predicted height value of the image area corresponding to the target object in the Nth frame image after the current frame image can be predicted based on the high change speed and high change acceleration of the image area corresponding to the target object in the current frame image. The position information of the target object in the current frame image can be determined as shown in formulas (11) to (12):

[0114]

[0115]

[0116] in, Indicates the first predicted width value of the corresponding image area of ​​the target object in the i+Nth frame image, Indicates the first predicted height value of the corresponding image area of ​​the target object in the i+Nth frame image. The first predicted size information may include and

[0117] Step 410: Using the first predicted region information as the observed value, the first image region information and the observed value are processed by a filter to predict the second predicted region information of the target object in the Nth frame image after the current frame image.

[0118] In one embodiment, Figure 7 As shown, the step takes the first predicted area information as the observation value, processes the first image area information and the observation value through a filter, and predicts the second predicted area information of the target object in the Nth frame image after the current frame image, which may include steps 702 to 704.

[0119] Step 702: Calculate the first image region information through a prediction model to obtain a filter prediction value.

[0120] The filter may include a prediction model and a correction model. In the embodiment of the present application, the filter may include a Kalman filter. The Kalman filter can infer the next step of the system based on the predicted value and observed value of the system and the correlation between the predicted value and the noise. It is an efficient autoregressive filter. The prediction model can be used to predict the system state at the next moment, and the correction model can correct the predicted system state at the next moment based on the predicted noise, observed noise, and combined with the observed value.

[0121] In one embodiment, the step of calculating the first image area information through a prediction model to obtain a filter prediction value may include: obtaining the change speed corresponding to the target object in the current frame image, calculating the first image area information and the change speed based on the prediction model to obtain the estimated state value corresponding to the target object at the next moment, and using the estimated state value as the filter prediction value.

[0122] The prediction model can be constructed under the assumption that the target object changes at a uniform speed, and the state corresponding to the target object at the next moment is estimated based on the change speed corresponding to the target object in the current frame image. In some embodiments, the current state value corresponding to the target object at the current moment can be determined based on the first image area information and the change speed. The current state value can be used to describe the system state of the target object in the current frame image, and the current state value is calculated through the state transfer matrix of the prediction model to obtain the estimated state value corresponding to the target object at the next moment. The number of frames N can be delayed as a unit time, and the estimated state value corresponding to the target object at the next moment can be used to describe the predicted state of the target object in the Nth frame image after the current frame image.

[0123] As a specific implementation, the estimated state value corresponding to the target object at the next moment may be calculated according to the following formulas (13) to (14).

[0124] c i+N =Ac i -Bu i+N (13);

[0125] P i+N =AP i A T +Q (14);

[0126] Among them, A is the state transfer matrix, B is the input control matrix, and u i+N is the external control quantity, P is the error matrix, and Q is the predicted noise covariance matrix. i represents the current state value (i.e., the state value corresponding to the i-th frame image), c i+NRepresents the estimated state value corresponding to the next moment (ie, the estimated state value corresponding to the i+Nth frame image).

[0127] In the embodiment of the present application, assuming that the target object moves at a uniform speed and is not disturbed, the state transfer matrix u i+N = 0. Predicted noise covariance matrix Among them, q is the debugging parameter.

[0128] In some embodiments, the first image area information includes position information, the change speed includes the position change speed, and the filter prediction value includes the filter prediction position information. Then, the current state value of the target object It can be used to describe the position state of the target object in the i-th frame image, based on the current state value c i Get the estimated state value c at the next moment i+N , the c i+N It can be used to describe the estimated position state of the target object in the i+Nth frame image, that is, as the filter predicts the position information.

[0129] In some embodiments, the first image region information includes size information, the change speed includes size change speed, and the filter prediction value includes filter prediction size information. Then, the current state value of the target object It can be used to describe the size state of the corresponding image area of ​​the target object in the i-th frame image, which can be based on the current state value c i Get the estimated state value c at the next moment i+N , the c i+N It can be used to describe the estimated size state of the corresponding image area of ​​the target object in the i+Nth frame image, that is, as the filter prediction size information.

[0130] Step 704, using the first prediction region information as the observation value of the correction model, and using the observation value in the correction model to correct the filter prediction value, to obtain the second prediction region information of the target object in the Nth frame image after the current frame image.

[0131] In some embodiments, the first prediction region information can be input as an observation value into the correction model, and the filter prediction value can be corrected by the noise matrix and the observation value of the correction model to obtain a corrected filter prediction value, and the second prediction region information can be obtained based on the corrected filter prediction value. Optionally, since the filter prediction value is a state value, the value of the target dimension in the corrected filter prediction value can be taken as the second prediction region information of the target object in the Nth frame image after the current frame image. For example, the values ​​of the first two dimensions in the corrected filter prediction value can be taken as the second prediction region information of the target object in the Nth frame image after the current frame image.

[0132] The noise matrix may include a measured noise covariance matrix, a predicted noise covariance matrix, etc. As a specific implementation, the filter prediction value may be corrected according to the following formulas (15) to (17):

[0133] K i+N =P i+N H T (HP i+N H T +R) -1 (15);

[0134]

[0135] P i+N =(IK i+N H)P i+N (17);

[0136] Where K is the Kalman gain, z i+N is the estimated observation value corresponding to the next moment (in the embodiment of the present application, it can be the first prediction area information), H represents the observation matrix, R represents the measurement noise covariance matrix, and I is the unit matrix. Represents the corrected filter prediction value.

[0137] In the embodiment of the present application, the observation matrix Measurement Error Matrix Among them, r is the debugging parameter.

[0138] While using the state transfer matrix to calculate the current state value and obtain the estimated state value corresponding to the target object at the next moment, the error matrix P at the next moment can be predicted, and the Kalman gain K at the next moment can be updated based on the predicted error matrix P at the next moment, and the update is continuously performed iteratively.

[0139] In some embodiments, the filter prediction value includes the filter prediction position information, the first prediction area information may include the first prediction position information, then the observation value corresponding to the next moment Based on the current state value of the target object The estimated state value c at the next moment is obtained i+N , this observation can be used to i+N After correction, the dimension of the corrected filter prediction position information is equal to the current state value c i Keep consistent, therefore, the values ​​of the first two dimensions can be taken as the second predicted position information output by the Kalman filter.

[0140] In some embodiments, the filter prediction value includes filter prediction size information, and the first prediction region information may include first prediction size information. Then, the observation value corresponding to the next moment is Based on the current state value of the target object The estimated state value c at the next moment is obtained i+N , this observation can be used to i+N After correction, the dimension of the corrected filter prediction position information is equal to the current state value c i Keep consistent, therefore, the values ​​of the first two dimensions can be taken as the second predicted size information output by the Kalman filter. It can be understood that the target dimension can be adjusted according to the matrix composition of the state value in actual use, and the first two dimensions are only an exemplary implementation and are not used to limit the embodiments of this application.

[0141] Figure 8 FIG. 1 is a schematic diagram of using a filter to predict the Nth frame image of a target object after the current frame image in one embodiment. Figure 8 As shown, the position information and size information of the target object in the current frame image can be input into the Kalman prediction model. The Kalman prediction model obtains the predicted position information based on the position information of the target object in the current frame image. The Kalman prediction model obtains the predicted size information based on the size information of the target object in the current frame image. The first predicted position information and the first predicted size information of the target object in the Nth frame image after the current frame image can be input into the Kalman modification model as observation values. The Kalman modification model can correct the position information predicted by the Kalman prediction model according to the first predicted position information to obtain the second predicted position information of the target object in the Nth frame image after the current frame image. The Kalman modification model can correct the size information predicted by the Kalman prediction model according to the first predicted size information to obtain the second predicted size information of the target object in the Nth frame image after the current frame image.

[0142] Step 412: Control the focus module to perform a focus operation based on the second prediction area information.

[0143] The focus module may determine a focus area based on the second predicted position information and the second predicted size information predicted by the Kalman filter, and perform a focus operation.

[0144] In some embodiments, the above-mentioned N can also be a number of frames larger than the number of delayed frames. Then, when the camera captures the Nth frame image after the current frame image, the focus module can be controlled to perform a focusing operation based on the second predicted area information corresponding to the Nth frame image after the current frame image, thereby making the focus tracking process more flexible and accurate.

[0145] It should be noted that the position coordinates of the target object in the above embodiments may be the center coordinates of the corresponding image area of ​​the target object in the image, or may be the corner coordinates of the corresponding image area, etc., which are not limited here.

[0146] In an embodiment of the present application, based on the Kalman filtering method, combined with the speed and acceleration of the position and size changes, the position and size of the target object in the Nth frame image after the current frame image are predicted, so as to correct the focus area of ​​the focus module, so that when the target object is in a moving state, the target object can accurately fall into the focus area, thereby improving the focus accuracy, optimizing the problem of unclearness of the target object during movement, and improving the imaging effect of the target object.

[0147] Moreover, in the process of predicting the target object, if only the first-order derivative is used for prediction, that is, assuming that the target motion speed remains unchanged, the perception of motion changes is weak, and it is difficult to accurately predict the target with motion changes. If only the changing acceleration is used for prediction, that is, the second-order derivative is used for prediction, it is easily affected by mutations and noise. In the embodiment of the present application, the first prediction area information based on the changing acceleration is used as the observation value, the uniform speed is used as the filter prediction value of the Kalman filter, and the filter prediction value is corrected in combination with the noise matrix, so that more accurate prediction information can be obtained, and the accuracy of the prediction is improved.

[0148] In addition, in the embodiment of the present application, predictions are made separately from two dimensions: target position and size. While ensuring that the position prediction is accurate, the focus area is able to always include the target object through the prediction of scale changes, thereby effectively reducing the error of the focus area of ​​the target object and optimizing the focus tracking effect on the target object.

[0149] In other embodiments, in addition to the above-mentioned Kalman filtering method, other algorithms may also be used to predict the Nth frame image of the target object after the current frame image. For example, the optical flow method may also be used to predict the Nth frame image of the target object after the current frame image. The optical flow refers to the instantaneous speed of the target object at the pixel level in the imaging domain. The optical flow method is a method of calculating the optical flow using the speed of grayscale change of the corresponding pixel position of the target object in two consecutive frames of images. The optical flow method may include but is not limited to a gradient-based optical flow method, a matching-based optical flow method, and a phase-based optical flow method.

[0150] In some embodiments, taking the gradient-based optical flow method as an example, the corner points of the target object in the image area corresponding to the current frame image can be detected, and based on the assumption of spatial consistency and the least squares method, the movement vector of each corner point is calculated (the movement vector can be used to describe the movement direction and movement speed), and then the corresponding movement direction and movement speed of the target object are obtained based on the movement vector of each corner point, thereby predicting the position information of the target object in the Nth frame image after the current frame image.

[0151] Optionally, for predicting the size information of the target object in the Nth frame image after the current frame image, the depth information of the target object in the current frame image can be obtained, and the depth information can be used to indicate the distance between the target object and the camera. The depth information can be obtained by infrared ranging, laser ranging, binocular ranging, etc., which are not limited here. The depth change information of the target object in the historical frame image and the depth information of the current frame image can be used to calculate the depth change information, and the size change information of the target object in the Nth frame image after the current frame image can be predicted based on the depth change information, and then the predicted size information of the target object in the Nth frame image after the current frame image can be obtained based on the size change information.

[0152] In an embodiment of the present application, by predicting the position information and size information of the target object in the Nth frame image after the current frame image, the focus area in the Nth frame image after the current frame image can be accurately estimated, thereby improving the focus accuracy and ensuring the imaging effect of the target object.

[0153] like Fig. 9 As shown, in one embodiment, an auto-focusing device 900 is provided, which can be applied to the above electronic device. The auto-focusing device 900 may include an information determination module 910, a first prediction module 920, a second prediction module 930 and a control module 940.

[0154] The information determination module 910 is used to determine the first image region information corresponding to the target object in the current frame image.

[0155] The first prediction module 920 is used to predict first prediction region information of the target object in the Nth frame image after the current frame image according to the first image region information, where N is a positive integer.

[0156] The second prediction module 930 is used to use the first prediction area information as the observation value, process the first image area information and the observation value through a filter, and predict the second prediction area information of the target object in the Nth frame image after the current frame image.

[0157] In one embodiment, N is the number of delayed frames, and the number of delayed frames is determined according to the duration of the first prediction region information and the second prediction region information of the predicted target object in the Nth frame image after the current frame image.

[0158] The control module 940 is used to control the focus module to perform a focus operation based on the second prediction area information.

[0159] In an embodiment of the present application, the first image area information corresponding to the target object in the current frame image is determined, and the first predicted area information of the target object in the Nth frame image after the current frame image is predicted based on the first image area information. The first predicted area information is used as an observation value, and the first image area information and the observation value are processed by a filter to predict the second predicted area information of the target object in the Nth frame image after the current frame image, and the focus module is controlled to perform a focus operation based on the second predicted area information. By predicting the focus area of ​​the target object in the Nth frame image after the current frame image, and then focusing based on the focus area, the deviation between the tracking result and the actual position of the target object can be reduced, and the accuracy of the autofocus is improved. Moreover, the first predicted area information is used as the observation value of the filter for prediction, which improves the accuracy of the prediction result and the imaging effect of the target object.

[0160] In one embodiment, the above-mentioned auto-focusing device 900 , in addition to the information determination module 910 , the first prediction module 920 , the second prediction module 930 , and the control module 940 , further includes an object determination module.

[0161] The object determination module is used to determine the target object from the initial frame image and obtain the initial image area of ​​the target object in the initial frame image.

[0162] The information determination module 910 is also used to match the current frame image with the initial image area, determine the target area in the current frame image that has the highest similarity with the initial image area, and obtain the first image area information corresponding to the target object in the current frame image based on the target area.

[0163] In one embodiment, the first prediction module 920 includes a speed calculation unit and a first prediction unit.

[0164] The speed calculation unit is used to calculate the change speed information corresponding to the target object in the current frame image based on the second image area information corresponding to the target object in the historical frame image and the first image area information. The historical frame image is an image collected by the camera before the current frame image.

[0165] In one embodiment, the frame number difference between the historical frame image and the current frame image is positively correlated with the delay time, and the delay time is determined based on the duration of the first prediction area information and the second prediction area information of the predicted target object in the Nth frame image after the current frame image.

[0166] In one embodiment, the frame number difference between the historical frame image and the current frame image is an integral multiple of N.

[0167] In one embodiment, the change speed includes the position change speed and the position change acceleration, and the first prediction area information includes the first prediction position information; and / or

[0168] The change speed includes the size change speed and the size change acceleration, and the first prediction area information includes the first prediction size information.

[0169] The first prediction unit is used to predict first prediction area information of the target object in the Nth frame image after the current frame image according to the change speed information.

[0170] In one embodiment, the filter includes a prediction model and a correction model.

[0171] The second prediction module 930 includes a second prediction unit and a correction unit.

[0172] The second prediction unit is used to calculate the first image area information through a prediction model to obtain a filter prediction value.

[0173] In one embodiment, the second prediction unit is also used to obtain the change speed corresponding to the target object in the current frame image, calculate the first image area information and the change speed based on the prediction model, obtain the estimated state value corresponding to the target object at the next moment, and use the estimated state value as the filter prediction value.

[0174] In one embodiment, the second prediction unit is also used to determine the current state value corresponding to the target object at the current moment based on the first image area information and the change speed, and calculate the current state value through the state transfer matrix of the prediction model to obtain the estimated state value corresponding to the target object at the next moment.

[0175] In one embodiment, the first image area information includes position information, the change speed includes position change speed, and the filter prediction value includes filter prediction position information; and / or

[0176] The first image region information includes size information, the change speed includes size change speed, and the filter prediction value includes filter prediction size information.

[0177] The correction unit is used to use the first prediction area information as the observation value of the correction model, and use the observation value in the correction model to correct the filter prediction value to obtain the second prediction area information of the target object in the Nth frame image after the current frame image.

[0178] In one embodiment, the correction unit is also used to correct the filter prediction value by correcting the noise matrix and observation value of the model to obtain a corrected filter prediction value, and take the numerical value of the target dimension in the corrected filter prediction value as the second prediction area information of the target object in the Nth frame image after the current frame image.

[0179] In an embodiment of the present application, based on the Kalman filtering method, combined with the speed and acceleration of the position and size changes, the position and size of the target object in the Nth frame image after the current frame image are predicted, so as to correct the focus area of ​​the focus module, so that when the target object is in a moving state, the target object can accurately fall into the focus area, thereby improving the focus accuracy, optimizing the problem of unclearness of the target object during movement, and improving the imaging effect of the target object.

[0180] Moreover, in the process of predicting the target object, if only the first-order derivative is used for prediction, that is, assuming that the target motion speed remains unchanged, the perception of motion changes is weak, and it is difficult to accurately predict the target with motion changes. If only the changing acceleration is used for prediction, that is, the second-order derivative is used for prediction, it is easily affected by mutations and noise. In the embodiment of the present application, the first prediction area information based on the changing acceleration is used as the observation value, the uniform speed is used as the filter prediction value of the Kalman filter, and the filter prediction value is corrected in combination with the noise matrix, so that more accurate prediction information can be obtained, and the accuracy of the prediction is improved.

[0181] In addition, in the embodiment of the present application, predictions are made separately from two dimensions: target position and size. While ensuring that the position prediction is accurate, the focus area is able to always include the target object through the prediction of scale changes, thereby effectively reducing the error of the focus area of ​​the target object and optimizing the focus tracking effect on the target object.

[0182] Fig.10 FIG. 1 is a structural block diagram of an electronic device in an embodiment. Fig.10 As shown, the electronic device 1000 may include one or more of the following components: a processor 1010, and a memory 1020 coupled to the processor 1010, wherein the memory 1020 may store one or more computer programs, and the one or more computer programs may be configured to implement the methods described in the above embodiments when executed by one or more processors 1010.

[0183] The processor 1010 may include one or more processing cores. The processor 1010 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1020, and calling data stored in the memory 1020. Optionally, the processor 1010 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1010 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1010, but may be implemented separately through a communication chip.

[0184] The memory 1020 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1020 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the electronic device 1000 during use, etc.

[0185] It is understandable that the electronic device 1000 may include more or fewer structural elements than those in the above structural block diagram, for example, a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, a sensor, etc., and is not limited here.

[0186] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the method described in the above embodiment when executed by a processor.

[0187] An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program can be executed by a processor to implement the methods described in the above embodiments.

[0188] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a ROM, etc.

[0189] As used herein, any reference to memory, storage, database, or other medium may include nonvolatile and / or volatile memory. Suitable nonvolatile memory may include ROM, programmable ROM (Programmable ROM, PROM), erasable PROM (Erasable PROM, EPROM), electrically erasable PROM (Electrically Erasable PROM, EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. By way of illustration and not limitation, RAM may be in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM) and direct memory bus dynamic RAM (DRDRAM).

[0190] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0191] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0192] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0194] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0195] The above is a detailed introduction to an autofocus method, device, electronic device, and computer-readable storage medium disclosed in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those of ordinary skill in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An automatic focusing method, characterized in that: include: Determine first image region information corresponding to the target object in the current frame image; Predicting first predicted region information of the target object in an N-th frame image after the current frame image according to the first image region information, where N is a positive integer and N is the number of delayed frames, and the number of delayed frames is determined according to the duration of predicting the first predicted region information and the second predicted region information of the target object in the N-th frame image after the current frame image; The first predicted region information is used as an observation value, and the first image region information and the observation value are processed by a filter to predict and obtain second predicted region information of the target object in the Nth frame image after the current frame image; The focusing module is controlled to perform a focusing operation based on the second prediction area information.

2. The method according to claim 1, characterized in that The predicting, according to the first image region information, first predicted region information of the target object in the Nth frame image after the current frame image comprises: Calculating the change speed information corresponding to the target object in the current frame image according to the second image region information corresponding to the target object in the historical frame image and the first image region information, wherein the historical frame image is an image captured by a camera before the current frame image; Predict first predicted region information of the target object in an N-th frame image after the current frame image according to the change speed information.

3. The method according to claim 2, characterized in that The frame number difference between the historical frame image and the current frame image is positively correlated with the delay time, and the delay time is determined based on the duration of the first predicted area information and the second predicted area information of the target object in the Nth frame image after the current frame image.

4. The method according to claim 3, characterized in that The frame number difference between the historical frame image and the current frame image is an integral multiple of N.

5. The method according to any one of claims 2 to 4, characterized in that: The change speed includes a position change speed and a position change acceleration, and the first prediction area information includes first prediction position information; and / or The change speed includes a size change speed and a size change acceleration, and the first predicted area information includes first predicted size information.

6. The method according to claim 1, characterized in that The filter includes a prediction model and a correction model; the first prediction region information is used as an observation value, the first image region information and the observation value are processed by the filter, and the second prediction region information of the target object in the Nth frame image after the current frame image is predicted, including: Calculating the first image area information by using the prediction model to obtain a filter prediction value; The first prediction region information is used as the observation value of the correction model, and the filter prediction value is corrected by using the observation value in the correction model to obtain the second prediction region information of the target object in the Nth frame image after the current frame image.

7. The method according to claim 6, characterized in that The calculating the first image area information by using the prediction model to obtain a filter prediction value includes: Obtaining the change speed of the target object in the current frame image; The first image area information and the change speed are calculated based on the prediction model to obtain an estimated state value corresponding to the target object at the next moment, and the estimated state value is used as a filter prediction value.

8. The method according to claim 7, characterized in that The calculating the first image area information and the change speed based on the prediction model to obtain the estimated state value corresponding to the target object at the next moment includes: Determine a current state value corresponding to the target object at a current moment according to the first image area information and the change speed; The current state value is calculated through the state transfer matrix of the prediction model to obtain the estimated state value corresponding to the target object at the next moment.

9. The method according to claim 7 or 8, characterized in that: The first image area information includes position information, the change speed includes position change speed, and the filter prediction value includes filter prediction position information; and / or The first image region information includes size information, the change speed includes size change speed, and the filter prediction value includes filter prediction size information.

10. The method according to claim 6, characterized in that The filter prediction value is corrected by using the observation value in the correction model to obtain second prediction area information of the target object in the Nth frame image after the current frame image, including: Correcting the filter prediction value by using the noise matrix of the correction model and the observation value to obtain a corrected filter prediction value; The value of the target dimension in the modified filter prediction value is taken as the second prediction area information of the target object in the Nth frame image after the current frame image.

11. The method according to claim 1, characterized in that: Before determining the first image region information corresponding to the target object in the current frame image, the method further includes: Determine a target object from an initial frame image, and obtain an initial image region of the target object in the initial frame image; The determining of first image region information corresponding to the target object in the current frame image includes: Matching the current frame image with the initial image region to determine the target region in the current frame image that has the highest similarity to the initial image region; The first image region information corresponding to the target object in the current frame image is obtained according to the target region.

12. An automatic focusing device, characterized in that: include: An information determination module, used to determine first image region information corresponding to the target object in the current frame image; A first prediction module, configured to predict first prediction region information of the target object in an N-th frame image after the current frame image according to the first image region information, wherein N is a positive integer and N is a number of delayed frames, and the number of delayed frames is determined according to a duration of predicting the first prediction region information and the second prediction region information of the target object in an N-th frame image after the current frame image; A second prediction module is used to use the first prediction area information as an observation value, process the first image area information and the observation value through a filter, and predict the second prediction area information of the target object in the Nth frame image after the current frame image; A control module is used to control the focus module to perform a focus operation based on the second prediction area information.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images