Camera focus method

Abstract

The present application discloses a camera focusing method, device and computer-readable storage medium, the method comprising: inputting a first original image and a second original image captured by a camera into a first channel and a second channel for processing, respectively, wherein the first original image and the second original image have an intersection in physical space; extracting common features for clarity calculation in the processed first original image and the processed second original image; determining the matching degree between the first common feature and the second common feature of the same type; wherein there are multiple common features, the common feature for clarity calculation in the first original image is the first common feature, and the common feature for clarity calculation in the second original image is the second common feature; determining the relative clarity of the second original image relative to the first original image according to a preset weight and corresponding matching degree of each common feature; focusing based on the relative clarity, the present application achieves accurate focusing.

Description

Technical Field

[0001] The present application relates to the field of automatic focusing technology, and in particular to a camera focusing method, device, and computer-readable storage medium. Background Art

[0002] With the rapid development of camera technology, people have higher and higher requirements for camera clarity. Based on this, higher requirements are also placed on the camera focusing process.

[0003] At present, the camera focus adopts a clarity evaluation algorithm based on a single image. Specifically, the focusing lens group is moved to obtain the image at the corresponding position; the clarity of the image at the corresponding position is calculated using the evaluation algorithm (y=f(x)), and a clarity evaluation curve between the image and clarity at the corresponding position is obtained. The clearest image position is determined based on the clarity evaluation curve, and the lens is moved to the clearest image position to complete the focus (the farther the focusing lens group is from the clearest image position, the blurrier the image).

[0004] The above-mentioned focusing method can only use the same image area (focusing lens group) for clarity calculation. When the image content of the previous and next frames changes, it will interfere with the evaluation algorithm, causing the clarity evaluation curve to no longer be monotonous and smooth, but to jump or form local peaks, resulting in focusing errors or focusing failures. Summary of the Invention

[0005] In view of this, embodiments of the present application provide a camera focusing method, device, and computer-readable storage medium, aiming to solve the technical problem of the existing focusing method being difficult to accurately focus when the environment changes dynamically.

[0006] An embodiment of the present application provides a camera focusing method, the method comprising:

[0007] Inputting a first original image and a second original image captured by a camera into a first channel and a second channel respectively for processing, wherein the first original image and the second original image have an intersection in physical space;

[0008] extracting common features for clarity calculation from the processed first original image and the processed second original image;

[0009] Determining a degree of matching between a first common feature and a second common feature of the same type; wherein the common features are multiple, the common feature used for clarity calculation in the first original image is the first common feature, and the common feature used for clarity calculation in the second original image is the second common feature;

[0010] determining a relative clarity of the second original image relative to the first original image based on a preset weight of each common feature and the corresponding matching degree;

[0011] Focusing is performed based on the relative sharpness.

[0012] In a possible implementation of the present application, the step of inputting the first original image and the second original image captured by the camera into the first channel and the second channel respectively for processing includes:

[0013] Input the first original image into the first channel, and input the second original image into the second channel;

[0014] The common features used for clarity calculation in the first original image after the extraction process and the second original image after the processing include:

[0015] Extracting common features for clarity calculation from the first image and the second image.

[0016] In a possible implementation manner of the present application, the step of extracting common features for clarity calculation in the first image and the second image includes:

[0017] Determining feature templates, wherein element values of each feature template are obtained based on preset annotation data and learned through a preset algorithm model, and there are multiple feature templates;

[0018] Performing feature extraction on each image according to the preset feature template to obtain a set of first feature maps for the first image and a set of second feature maps for the second image, wherein the first feature maps and the second feature maps are numbered in the same order, each feature map is composed of feature values of a plurality of common features, and the feature values are calculated values corresponding to regions in each image that undergo convolution calculation with the feature template;

[0019] The step of determining the degree of matching between the first common feature and the second common feature of the same type comprises:

[0020] For common features of the same type, determine a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number. The matching first feature value is the maximum value of the corresponding block area of the first feature map and the second feature map;

[0021] The matching degree between each pair of first feature values is determined as the matching degree between the first common feature and the second common feature of the same type.

[0022] In a possible implementation of the present application, the step of determining, for common features of the same type, a matching first feature value from a first feature map and a second feature map corresponding to the same sequence number includes:

[0023] For common features of the same type, determining a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number;

[0024] Each feature map after each feature extraction is scaled and features are re-extracted to obtain multiple sets of feature submaps of different sizes, where the smaller the size of the feature map, the lower the resolution of the feature map;

[0025] For common features of the same type, a matching second eigenvalue is determined from two feature sub-images with the same resolution. The matching second eigenvalue is based on the maximum value of the corresponding block areas of the two feature sub-images with the same resolution.

[0026] The step of determining the matching degree between each pair of first feature values as the matching degree between the first common feature and the second common feature of the same type includes:

[0027] The matching degree between each pair of first eigenvalues and the matching degree between each pair of second eigenvalues are both determined as the matching degree between the first common feature and the second common feature of the same type.

[0028] In a possible implementation of the present application, for common features of the same type, determining a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number, where the matching first feature value is the maximum value of the corresponding block areas of the first feature map and the second feature map, includes:

[0029] For common features of the same type, a matching first feature value is determined from the first feature map and the second feature map corresponding to the same sequence number, and the matching first feature value is the maximum value of the block area corresponding to the first feature map and the second feature map, wherein, when the common features of the same type are associated with a moving object, the size of the block area is correlated with the moving speed of the moving object.

[0030] In a possible implementation manner of the present application, the step of determining the relative clarity of the second original image relative to the first original image based on the preset weight of each common feature and the corresponding matching degree includes:

[0031] According to the preset weight of each common feature and the corresponding matching degree, if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is greater than a preset value, then determining that the second original image is clearer than the first original image;

[0032] If the relative clarity of the second original image in the second channel relative to the first original image in the first channel is less than the preset value, determining that the first original image is clearer than the second original image;

[0033] If the relative clarity of the second original image in the second channel relative to the first original image in the first channel is equal to the preset value, the clarity between the first original image and the second original image cannot be distinguished.

[0034] An embodiment of the present application provides a camera focusing method, the method comprising:

[0035] Focusing the camera, selecting a first image and a second image, wherein the second image includes a portion of identical image content and a portion of different image content, wherein the portion of different image content is in response to a dynamic disturbance imposed by a dynamic object;

[0036] Inputting the first image via a first channel and inputting the second image via a second channel;

[0037] performing grayscale processing and normalization processing on the input first image and the second image;

[0038] Performing feature extraction on the normalized first image and the second image, comprising:

[0039] In response to a plurality of preset feature templates, matching is performed on the normalized first image and the second image to determine a plurality of first feature maps and a plurality of second feature maps that match the plurality of preset feature templates, wherein any pixel value in the first feature map is a matrix product value of the preset feature template and the normalized first image, and any pixel value in the second feature map is a matrix product value of the preset feature template and the normalized second image;

[0040] determining a clarity value of the second image relative to the first image, the clarity value being weighted by a difference between the second feature map and the associated first feature map;

[0041] A focus curve of the camera is determined according to the clarity value.

[0042] In a possible implementation manner of the present application, the first image and the second image are defined as images in an image sequence generated during the process of moving the focusing lens group from the farthest end to the near end.

[0043] In a possible implementation of the present application,

[0044] Any pixel value in the first feature map W(i,j) Represents the element value of the feature template at (i, j), I (x-1+i,y-1+j) Represents the pixel value of a picture of the same size as the feature template that is captured from the normalized first image.

[0045] In a possible implementation of the present application, the clarity value of the second image relative to the first image is obtained by subtracting each pair of matching feature values, multiplying them by different weights according to feature type and matching status, and taking the average of the accumulated sums as the final result of measuring relative clarity to suppress interference from dynamic factors, wherein:

[0046] The clarity value of the second image relative to the first image

[0047] Among them, Image2Feature i Value and Image1Feature i Value is the matching feature value pair of the second image and the first image corresponding to the i-th matching feature, and Image2Feature i Value-Image1Feature i Value is the matching status of the i-th matching feature, w i is the influence weight of the i-th matching feature on the clarity.

[0048] The present application also provides a camera focusing device, the device comprising:

[0049] An input module, configured to input a first original image and a second original image captured by a camera into a first channel and a second channel respectively for processing, wherein the first original image and the second original image have an intersection in physical space;

[0050] a feature extraction module, configured to extract common features for clarity calculation from the processed first original image and the processed second original image;

[0051] a feature matching module, configured to determine a degree of matching between a first common feature and a second common feature of the same type; wherein the common features are multiple, the common feature used for clarity calculation in the first original image is the first common feature, and the common feature used for clarity calculation in the second original image is the second common feature;

[0052] a clarity calculation module, configured to determine a relative clarity of the second original image relative to the first original image based on a preset weight of each common feature and the corresponding matching degree;

[0053] A result output module is used to focus based on the relative clarity.

[0054] In a possible implementation of the present application, the input module is configured to: input the first original image into the first channel, and input the second original image into the second channel; the feature extraction module includes: an extraction unit configured to extract common features for clarity calculation in the first image and the second image;

[0055] And / or the extraction unit is used to implement: determining a feature template, wherein the element value of each feature template is obtained based on preset annotation data and learned by a preset algorithm model, and there are multiple feature templates; performing feature extraction on each image according to the preset feature template to obtain a set of first feature maps of the first image and a set of second feature maps of the second image, wherein the first feature maps and the second feature maps are numbered in the same order, and each feature map is composed of feature values of multiple common features, and the feature values are calculated values corresponding to the area in each image that is convolved with the feature template; the feature matching module is used to determine, for the same type of common features, the matching first feature value from the first feature map and the second feature map corresponding to the same sequence number, the matching first feature value being the maximum value of the corresponding block area of the first feature map and the second feature map; and determining the matching degree between each pair of first feature values as the matching degree between the first common feature and the second common feature of the same type;

[0056] And / or the feature matching module is configured to: determine, for a common feature of the same type, a matching first feature value from a first feature map and a second feature map corresponding to the same sequence number, wherein the matching first feature value is a maximum value of corresponding block areas in the first feature map and the second feature map, wherein, when the common feature of the same type is associated with a moving object, a size of the block area is associated with a moving speed of the moving object;

[0057] And / or the clarity calculation module includes: a first determination unit, used to determine, based on a preset weight of each common feature and the corresponding matching degree, that the second original image is clearer than the first original image if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is greater than a preset value; a second determination unit, used to determine that the first original image is clearer than the second original image if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is less than the preset value; and a third determination unit, used to determine that the clarity between the first original image and the second original image cannot be distinguished if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is equal to the preset value.

[0058] The present invention provides a camera focusing device, which is used to achieve:

[0059] Focusing the camera, selecting a first image and a second image, wherein the second image includes a portion of identical image content and a portion of different image content, wherein the portion of different image content is in response to a dynamic disturbance imposed by a dynamic object;

[0060] Inputting the first image via a first channel and inputting the second image via a second channel;

[0061] performing grayscale processing and normalization processing on the input first image and the second image;

[0062] Performing feature extraction on the normalized first image and the second image, comprising:

[0063] In response to a plurality of preset feature templates, matching is performed on the normalized first image and the second image to determine a plurality of first feature maps and a plurality of second feature maps that match the plurality of preset feature templates, wherein any pixel value in the first feature map is a matrix product value of the preset feature template and the normalized first image, and any pixel value in the second feature map is a matrix product value of the preset feature template and the normalized second image;

[0064] determining a clarity value of the second image relative to the first image, the clarity value being weighted by a difference between the second feature map and the associated first feature map;

[0065] A focus curve of the camera is determined according to the clarity value.

[0066] In a possible implementation manner of the present application, the device is used to implement: the first image and the second image are defined as images in an image sequence generated during the process of moving the focusing lens group from the farthest end to the near end.

[0067] In a possible implementation manner of the present application, the device is used to implement:

[0068] Any pixel value in the first feature map W (i,j) Represents the element value of the feature template at (i, j), I (x-1+i,y-1+j) Represents the pixel value of a picture of the same size as the feature template that is captured from the normalized first image.

[0069] In one possible implementation of the present application, the apparatus is configured to implement: a clarity value of the second image relative to the first image = subtracting each pair of matching feature values, multiplying them by different weights according to feature type and matching status, and taking the average of the accumulated sums as the final result of measuring relative clarity to suppress interference from dynamic factors, wherein:

[0070] The clarity value of the second image relative to the first image

[0071] Among them, Image2Feature i Value and Image1Feature i Value is the matching feature value pair of the second image and the first image corresponding to the i-th matching feature, and Image2Feature i Value-Image1Feature i Value is the matching status of the i-th matching feature, w i is the influence weight of the i-th matching feature on the clarity.

[0072] The present application also provides a camera focusing device, which is a physical node device. The camera focusing device includes: a memory, a processor, and a program of the camera focusing method stored in the memory and runnable on the processor. When the program of the camera focusing method is executed by the processor, the steps of the camera focusing method described above can be implemented.

[0073] To achieve the above object, a computer-readable storage medium is further provided, wherein a camera focus program is stored on the computer-readable storage medium, and when the camera focus program is executed by a processor, the steps of any of the above-mentioned camera focus methods are implemented.

[0074] The present application provides a camera focusing method, device and computer-readable storage medium. Compared with the prior art in which the camera uses a single-image clarity evaluation algorithm to focus, which causes easy focusing failure, in the present application, a first original image and a second original image captured by the camera are input into a first channel and a second channel respectively for processing, wherein the first original image and the second original image have an intersection in physical space; common features for clarity calculation are extracted from the processed first original image and the processed second original image; the matching degree between the first common feature and the second common feature of the same type is determined; wherein there are multiple common features, the common feature used for clarity calculation in the first original image is the first common feature, and the common feature used for clarity calculation in the second original image is the second common feature; according to the preset weight of each common feature and the corresponding matching degree, the relative clarity of the second original image relative to the first original image is determined; and focusing is performed based on the relative clarity. In the present application, since the clarity evaluation algorithm of a single image is no longer used to calculate the clarity, the common and matching features in the two frames of images (corresponding to the detection of the same physical object) are used to determine the relative clarity of the second original image relative to the first original image. That is, since the same physical object is detected in the two frames of images, the clarity of the two frames of images can be accurately detected. Moreover, since two frames of images are compared instead of a single frame of images, it is considered that the compared images themselves are different, and therefore are less affected by scene changes, which can effectively suppress the interference of dynamic factors, thereby improving the focusing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a flow chart of a first embodiment of the camera focusing method of the present application;

[0076] Figure 2 This is a flow chart of another embodiment of the camera focusing method of the present application;

[0077] Figure 3 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application;

[0078] Figure 4 This is a flow chart of the auto-focus process involved in the camera focusing method of this application;

[0079] Figure 5 A flow chart of a clarity evaluation curve involved in the camera focusing method of this application;

[0080] Figure 6 A schematic diagram of a scenario in which a moving object interferes with a clarity evaluation curve in the camera focusing method of this application;

[0081] Figure 7A schematic diagram of a scenario in which device jitter, involved in the camera focusing method of this application, interferes with the clarity evaluation curve;

[0082] Figure 8 This is a schematic diagram of the algorithm modules involved in the camera focusing method of this application;

[0083] Figure 9 This is a schematic diagram of feature extraction calculation involved in the camera focusing method of this application;

[0084] Figure 10 These are feature subgraphs of different sizes involved in the camera focusing method of this application;

[0085] Figure 11 This is a schematic diagram of feature matching involved in the camera focusing method of this application;

[0086] Figure 12 A schematic diagram of feature matching failure involved in the camera focusing method of this application;

[0087] Figure 13 This is a schematic diagram of the data annotation involved in the camera focusing method of this application. DETAILED DESCRIPTION

[0088] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0089] The present application provides a camera focusing method. In one embodiment of the camera focusing method of the present application, referring to Figure 1 , the method comprising:

[0090] Step S10: inputting a first original image and a second original image captured by a camera into a first channel and a second channel respectively for processing, wherein the first original image and the second original image have an intersection in physical space;

[0091] Step S20, extracting common features for clarity calculation from the processed first original image and the processed second original image;

[0092] Step S30, determining a matching degree between a first common feature and a second common feature of the same type; wherein the common features are multiple, the common feature used for clarity calculation in the first original image is the first common feature, and the common feature used for clarity calculation in the second original image is the second common feature;

[0093] Step S40: determining the relative clarity of the second original image relative to the first original image based on the preset weight of each common feature and the corresponding matching degree;

[0094] Step S50: focusing based on the relative clarity.

[0095] This embodiment aims to improve focusing accuracy.

[0096] Specifically, in the present application, since the clarity evaluation algorithm of a single image is no longer used to calculate the clarity, the common and matching features in the two frames of images (corresponding to the detection of the same physical object) are used to determine the relative clarity of the second original image relative to the first original image. That is, since the same physical object is detected in the two frames of images, the clarity of the two frames of images can be accurately detected, and since two frames of images are compared instead of a single frame of images, they are less affected by scene changes, and the interference of dynamic factors can be effectively suppressed, thereby improving the focusing accuracy.

[0097] Furthermore, in the present application, since there is no restriction on a fixed area, the effective area of the image can be flexibly selected to participate in the calculation, avoiding interference from invalid areas, and thus further improving the focusing accuracy (in the prior art, the consistency of the previous and subsequent sequence image scenes must be considered, so the same area has to be used for calculation).

[0098] Furthermore, in the present application, since the first original image and the second original image are respectively grayscaled, standardized and scaled in the corresponding channels, and the scaled partial areas are respectively intercepted to obtain the first image and the second image, and the common features used for clarity calculation in the first image and the second image are extracted, wherein the process of standardizing the grayscale original image is: subtracting the pixel value of each pixel point in the single original image from the pixel mean of the single original image, and dividing it by the standard deviation corresponding to the single original image; the first image and the second image have an intersection in physical space, and are smaller in size than the corresponding original image, that is, the present application does not require the entire image to participate in the calculation, but performs focusing calculation on a partial area, which reduces the amount of calculation and improves focusing efficiency.

[0099] Furthermore, in the present application, feature extraction is performed on each image according to the preset feature template to obtain a set of first feature maps of the first image and a set of second feature maps of the second image. For common features of the same type, matching first feature values are determined from the first feature maps and the second feature maps corresponding to the same sequential numbers. The matching first feature value is the maximum value of the corresponding block areas of the first feature map and the second feature map, thereby making it possible to accurately determine the matching degree of the first image and the second image, and further, accurately determine the clarity to achieve accurate focus.

[0100] Furthermore, in the present application, when the common features of the same type are associated with a moving object, the size of the block area is correlated with the moving speed of the moving object. Since the size of the block area is correlated with the moving speed of the moving object, when matching features, there will be no mismatching caused by feature duplication due to intercepting an overly large feature map. In addition, it also avoids matching failure caused by excessive feature offset due to the intercepting area being too small.

[0101] Furthermore, in the present application, each feature map after each feature extraction is scaled and the features are re-extracted to obtain multiple groups of feature sub-maps of different sizes, and the matching degree between each pair of first eigenvalues and the matching degree between each pair of second eigenvalues are determined as the matching degree between the first common feature and the second common feature of the same type. That is, in the present application, feature matching is performed based on feature maps of multiple levels of different resolutions, that is, feature matching uses feature maps containing multiple resolutions, so that the feature map clipping area is more accurate (the feature map clipping area cannot be too large to avoid feature duplication, resulting in mismatching and affecting subsequent calculation results. But if the clipping area is too small and the feature offset is too large, it will cause matching failure), and the present application uses multi-level feature maps, which can take into account high-resolution and low-resolution features, making the algorithm more adaptable and making the focus more accurate.

[0102] Furthermore, in this application, relative clarity is calculated, and the regression problem is converted into a classification problem to avoid matching failure and interference problems caused by invalid features.

[0103] Furthermore, in this application, the preset weights of the shared features are determined when minimizing a preset loss function based on training data with preset classification labels. This minimization of the preset loss function is performed using a gradient descent algorithm, and the degree of matching is determined based on the feature difference between the first shared feature and the second shared feature. This application solves the problem of data being unable to be labeled by converting the regression problem into a classification problem.

[0104] As an example, it should be noted that: "autofocus" refers to the process in which a camera autonomously drives a transmission device and moves a lens group through sensor information to form a clear image of a target object on the sensor.

[0105] There are two types of autofocus: active focus and passive focus. Among them, contrast focus in passive focus is widely used in cameras because it only requires image information.

[0106] Specifically, the focusing lens group is moved to obtain an image at a corresponding position; the clarity of the image at the corresponding position is calculated using an evaluation algorithm (y=f(x)), and a clarity evaluation curve between the image at the corresponding position and the clarity is obtained, such as Figure 5 As shown, the clearest image position is determined based on the clarity evaluation curve, and the lens is moved to the clearest image position to complete the focus (the farther the focusing lens group is from the clearest image position, the blurrier the image).

[0107] Depend on Figure 4 It can be determined from the process in that contrast focusing depends on the clarity detection value of the sequence image, and the clarity detection value is usually calculated by three types of algorithms: grayscale gradient-based, frequency domain-based, and information entropy-based.

[0108] Among them, the grayscale gradient-based algorithm uses the characteristic that clear images have larger grayscale differences and measures whether the image is clear by calculating the grayscale gradient value of the image. The frequency domain-based algorithm uses the fact that clear images have more high-frequency signals and processes the image using a high-pass filter to retain the high-frequency components. The amount of high-frequency components is used as a measure of whether the image is clear. The information entropy uses the characteristic that clear images are more orderly and have smaller entropy than blurred images and evaluates whether the image is clear by calculating the information entropy of the image.

[0109] The commonality among existing clarity calculation methods is that they take an image as input and output a numerical value, completing the mapping from image to clarity. Therefore, these algorithms are also called clarity evaluation algorithms.

[0110] The formula is as follows:

[0111] y=f(x)

[0112] Here, input x is the image data, and output y is the clarity, also known as the clarity test value. The clearer the input image, the larger the output test value. The farther the focusing lens group is from the clearest image position, the blurrier the image. Therefore, by connecting the clarity test values corresponding to images at different positions, we can obtain a bell-shaped curve that decreases monotonically with focus, namely the clarity evaluation curve, as shown in the figure below. Figure 5 As shown, the existing focusing algorithm determines the direction and position of the clear point based on this characteristic of the clarity evaluation curve, and then completes the focusing.

[0113] The research and development background or motivation of this application is:

[0114] The inventors found that in the actual focusing process, the environment often changes dynamically, such as Figure 6 , Figure 7 As shown in the figure, the movement of the object or the jitter of the device itself, etc., these factors will cause the image content of the previous and next frames to change, which in turn affects the calculation results of the focusing function, making the evaluation curve no longer monotonous and smooth, but jumping or forming local peaks, which interferes with the evaluation algorithm and causes focusing failure.

[0115] like Figure 6As shown in the figure, there is no interference from moving objects during the capture process on the left side, and the evaluation curve is monotonous and smooth. On the right side, a vehicle intrudes, causing the curve to fluctuate and form a local peak.

[0116] like Figure 7 As shown in the figure, during the capture process, the device was disturbed by external factors such as wind and caused vibration, which caused the evaluation curve to jump.

[0117] The inventors discovered that existing evaluation algorithms have a common assumption, namely scene invariance. That is, existing evaluation algorithms assume that during the focusing process, only the clarity of the image changes, and the content of the image does not change. This makes this type of algorithm poorly resistant to dynamic interference when the environment changes.

[0118] In addition, the inventors found that the results of the existing evaluation algorithm are the average of the results of various parts of the image. If only a small area in the image contains valid information, and the rest is invalid or even interference information, the valid information will be obliterated, resulting in erroneous results. This is another reason why the existing evaluation algorithm has poor resistance to dynamic interference when the environment changes.

[0119] Based on the above findings, the inventors thought of using the common and matching features in the two frames of images (corresponding to the detection of the same physical object) to determine the relative clarity of the second original image relative to the first original image. That is, since the same physical object is detected in the two frames of images, the clarity of the two frames of images can be accurately detected. Moreover, since two frames of images are compared instead of a single frame of image, it is considered that the compared images themselves are different, and therefore are less affected by scene changes, which can effectively suppress the interference of dynamic factors, thereby improving the focusing accuracy.

[0120] Based on the above findings, the inventors also thought that the present application uses an image of a valid area, thereby avoiding erroneous results caused by the obliteration of valid information.

[0121] like Figure 8 As shown, the camera in this application consists of a data input module, a feature extraction module, a feature matching module, a clarity calculation module and a result output module.

[0122] The specific steps are as follows:

[0123] Step S10: inputting a first original image and a second original image captured by a camera into a first channel and a second channel respectively for processing, wherein the first original image and the second original image have an intersection in physical space;

[0124] As an example, the input module of the camera inputs the first original image and the second original image captured by the camera into the first channel and the second channel respectively for processing.

[0125] As an example, the data input module specifies the input specifications of the data and pre-processes the data.

[0126] As an example, the data input module has two channels, namely the first channel and the second channel, so the first original image and the second original image should be input strictly according to the corresponding channels. The same image has different input positions and the meaning of the output results is different.

[0127] For ease of explanation, image1 is used to refer to the first original image input into the first channel, and image2 is used to refer to the second original image input into the second channel.

[0128] As an example, the first original image and the second original image must contain the same physical object, that is, image1 and image2 must have an intersection in the physical space.

[0129] As an example, the areas of the first original image and the second original image account for more than 10% of the total image area, as shown below:

[0130] Simage1∩Simage2>10%*Simage1=10%*Simage2;

[0131] Among them, S x is the set of pixels of image x.

[0132] The step of inputting the first original image and the second original image captured by the camera into the first channel and the second channel for processing respectively includes:

[0133] Step S11, inputting the first original image into the first channel, and inputting the second original image into the second channel;

[0134] Step S12: grayscale, normalize, and scale the first original image and the second original image in the corresponding channels, and intercept the scaled partial areas to obtain a first image and a second image, wherein the first image and the second image intersect in physical space and are both smaller in size than the corresponding original image;

[0135] The first original image and the second original image are respectively grayscaled, standardized and scaled in the corresponding channels, and the scaled partial areas are respectively captured to obtain the first image and the second image. Since the image is grayscaled in this application, there is no restriction on the input original image, which broadens the application scenarios. In this application, the original image after the grayscale map is also standardized to weaken the impact of differences such as exposure on focus.

[0136] The process of the standardization processing is as follows: the pixel value of each pixel in the original image is subtracted from the pixel mean of all pixels in the original image to obtain the difference corresponding to each pixel in the original image, and the difference corresponding to each pixel is divided by the standard deviation corresponding to the original image;

[0137] Specifically, the process of normalizing the grayscaled original image is as follows: subtract the pixel mean of the single original image from the pixel value of each pixel in the single original image, and then divide the result by the standard deviation corresponding to the single original image. That is, in this embodiment, a single grayscaled image is used as an example for explanation. Assume that the pixel width of the grayscaled image is w, the height is h, and image(x, y) is the pixel value of the image at (x, y). The normalization process is as follows:

[0138]

[0139] in:

[0140]

[0141]

[0142] Where, mean and var are the pixel mean and variance respectively, srqrt(var) is the square root of var, which is the standard deviation, and image′(x,y) is the standardized image.

[0143] As an example, after normalization, the image is scaled (the specific scaling and interception size can be set and adjusted according to the camera hardware resources and the original image resolution).

[0144] As an example, a partial area of the processed image is captured as input to the feature extraction module.

[0145] Specifically, after the first original image and the second original image are grayscaled and standardized in the corresponding channels, they are scaled and partial area screenshots are performed accordingly to obtain a first image and a second image, wherein the first image and the second image have an intersection in physical space and are smaller in size than the corresponding original image.

[0146] Specifically, the scaling size adopted in this embodiment is to scale the original image to a size of 1920*1080, and then cut out the image to a size of 256*256 to obtain the first image and the second image.

[0147] Step S20, extracting common features for clarity calculation from the processed first original image and the processed second original image;

[0148] As an example, the common feature used for clarity calculation in the first original image is a first common feature, and the common feature used for clarity calculation in the second original image is a second common feature.

[0149] As an example, there are multiple common features, and thus, there are multiple first common features, and correspondingly, there are also multiple second common features.

[0150] As an example, a method for extracting common features for clarity calculation from the processed first original image and the processed second original image is:

[0151] The first original image and the second original image are subjected to convolution calculation processing using the same feature template.

[0152] Specifically, the feature template consists of a small square matrix, such as 3x3 or 5x5. The actual size of the feature template can be adjusted according to the algorithm effect. Each feature template is composed of the element value of the feature to be extracted, and there are multiple feature templates.

[0153] The method of extracting the common features for clarity calculation from the processed first original image and the processed second original image is as follows:

[0154] Use W to represent a feature template, W(i,j) represents the element value of the feature template at (i,j) (the element value of the feature to be extracted). For image image, when extracting features at (x,y), it is only necessary to cut out an area with the same size as the feature template centered at (x,y), and multiply and accumulate W(i,j) and image(x,y) one by one.

[0155] Specifically, the step of extracting common features for clarity calculation from the first image and the second image, and determining the degree of matching between the first common features and the second common features of the same type, includes:

[0156] Step M1, determining a feature template, wherein the element value of each feature template is obtained based on preset annotation data through learning of a preset algorithm model, and there are multiple feature templates;

[0157] Step M2: performing feature extraction on each image according to the preset feature template to obtain a set of first feature maps for the first image and a set of second feature maps for the second image, wherein the first feature maps and the second feature maps are numbered in the same order, and each feature map is composed of feature values of a plurality of common features, where the feature values are calculated values corresponding to the area in each image that is convolved with the feature template;

[0158] In this embodiment, W represents a feature template, W(i,j) represents the element value of the feature template at (i,j), and for an image image, when extracting features at (x,y), it is only necessary to cut out an area with the same size as the feature template centered at (x,y), and multiply and accumulate W(i,j) and image(x,y) one by one.

[0159] The same operation is performed on all positions (x, y) of the image, and the results are stored in the feature map (FeatureMap(x, y)), that is, according to the preset feature template, feature extraction is performed on each image to obtain a set of first feature maps of the first image and a set of second feature maps of the second image. In this embodiment, the first feature maps and the second feature maps are numbered in the same order, and each feature map is composed of multiple feature values of common features, and the feature values are calculated values corresponding to the area in each image that undergoes convolution calculation with the feature template.

[0160] In this embodiment, as follows Figure 9 As shown in FIG, the more similar the cutout area in the image is to W, the larger the calculation result is, that is, the calculation result reflects the degree of appearance of the W feature at (x, y) in the image, and the degree of appearance is expressed as a feature value.

[0161] The calculation process of the eigenvalue is as follows:

[0162]

[0163] Step S30, determining a matching degree between a first common feature and a second common feature of the same type; wherein the common features are multiple, the common feature used for clarity calculation in the first original image is the first common feature, and the common feature used for clarity calculation in the second original image is the second common feature;

[0164] After determining the common features, determine the feature value of the first common feature used for clarity detection in the first original image, determine the feature value of the second common feature used for clarity detection in the second original image, and determine the matching degree between the first common feature and the second common feature of the same type for calculating relative clarity.

[0165] The step of determining the degree of matching between the first common feature and the second common feature of the same type comprises:

[0166] Step L1: for common features of the same type, determine a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number, where the matching first feature value is the maximum value of the corresponding block area of the first feature map and the second feature map;

[0167] Step L2, determine the matching degree between each pair of first eigenvalues as the matching degree between the first common feature and the second common feature of the same type.

[0168] Specifically, in the feature extraction module, perform feature extraction on image1 and image2, and obtain a set of feature maps respectively (a set of first feature maps and a set of second feature maps).

[0169] Assume the number of feature maps is n, and number them in the same order. A set of first feature maps are respectively Image1FeatureMap1, Image1FeatureMap2, Image1FeatureMap n ; a set of second feature maps are respectively: Image2FeatureMap1, Image2FeatureMap2, Image2FeatureMap n .

[0170] Among them, the first feature map and the second feature map are numbered in the same order. The purpose of numbering in the same order is to facilitate pairing and matching. Pair the features with the same serial number, Figure 1 such as pair Image1FeatureMap i and Image2FeatureMap i to obtain n pairs of feature maps.

[0171] In this embodiment, it should be noted that for a certain feature i in image1, its coordinates are located at (x, y). Due to the influence of dynamic factors, it deviates to (x′, y′) in image2. Then a maximum value is formed at the (x, y) of the feature map Image1FeatureMap i , and mark it as Image1Feature i Value. Similarly, a maximum value is also formed at the (x′, y′) of Image2FeatureMap i , and mark it as Image2Feature i Value.

[0172] In this embodiment, for the common features of the same type, determine the matching first eigenvalues from the first feature map and the second feature map corresponding to the same order numbering. The matching first eigenvalue is the maximum value of the corresponding sub-block areas of the first feature map and the second feature map. The specific method is: set |x - x′| < h, |y - y′| < h, then at Image1Feature i Value and Image2FeatureMap iIn the example, we take (x,y) as the center and cut off the regions A1 and A2 with width and height of 2*w and 2*h. Assume that A1 and A2 correspond to the features in the image1 and image2 regions. i The feature is unique, then:

[0173] Image1Feature i Value=max(A1)

[0174] Image2Feature i Value=max(A2)

[0175] Since max(A x ) indicates A x The maximum value of all elements in , such as Figure 11 As shown, Image1Feature i Value and Image2Feature i Value is the first eigenvalue matched in image1 and image2, that is, the first eigenvalue matched is the maximum value of the corresponding block area based on the first feature map and the second feature map. The feature matching of the first image and the second image can be completed by matching the first eigenvalue.

[0176] Specifically, assuming that there is a car at (x, y) in image P1, and the common feature of the same type is the car feature (the corresponding feature template is responsible for detecting the car feature), then after convolution of the feature template and the image, a first feature map F1 is obtained (each pixel value in the first feature map reflects the possibility of the existence of a car at the corresponding position in the corresponding image). The first feature map F1 has the largest value at (x, y), and the values at other positions are relatively small.

[0177] As the car moves, in image P2, the car moves to (x+δ, y+δ), and the corresponding second feature map F2 has the largest value at (x+δ, y+δ), and the other positions are smaller.

[0178] Because the value at (x, y) in F1 and the value at (x+δ, y+δ) in F2 both correspond to car features, and both are the maximum values in their respective feature maps, the maximum value in the feature map is taken as a matching eigenvalue pair (a pair of first eigenvalues).

[0179] In this embodiment, it should be noted that for the common features of the same type, determining the matching first feature value from the first feature map and the second feature map corresponding to the same sequence number, where the matching first feature value is based on the maximum value of the corresponding block areas of the first feature map and the second feature map, includes:

[0180] For common features of the same type, a matching first feature value is determined from the first feature map and the second feature map corresponding to the same sequence number, and the matching first feature value is based on the maximum value of the corresponding block area of the first feature map and the second feature map, wherein, when the common features of the same type are associated with a moving object, the size of the block area is correlated with the moving speed of the moving object.

[0181] Specifically, assuming that the common features of the same type are car features, since cars move fast, the corresponding (x+δ, y+δ) and (x, y) are quite different. In order to avoid matching failure caused by excessive feature offset, it is necessary to intercept large block areas to improve the accuracy of clarity calculation. Suppose that the common features of the same type are pedestrian features. Since pedestrians move slowly, the corresponding (x+δ, y+δ) and (x, y) are not much different. In order to avoid waste of computing resources, relatively small block areas are intercepted to improve the speed of clarity calculation.

[0182] The difference between the two maximum eigenvalues reflects the degree of feature matching. The smaller the difference, the higher the feature matching degree, and vice versa.

[0183] It should be noted that for each pair of feature maps, multiple regions (for images) are sampled and intercepted at different locations based on their size for feature matching, resulting in multiple matching feature value pairs. The more samples are taken, the richer the information and the better the algorithm effect, but the greater the computational effort. In this embodiment, a balance is struck between the amount of information and the amount of computation based on actual needs.

[0184] In this embodiment, the first feature map and the second feature map corresponding to the same sequence number are a feature map pair, and the i-th feature value pair is recorded as (Image1Feature i Value, Image2Feature i Value), then Image1Feature1Value, Image1Feature2Value, Image1Feature n Value corresponds to Image1Feature2Value, Image1Feature n Value constitutes two n-dimensional feature value vectors, recorded as Image1Feature1ValueVecter and Image2Feature1ValueVecter.

[0185] Step S40: determining the relative clarity of the second original image relative to the first original image based on the preset weight of each common feature and the corresponding matching degree;

[0186] In this embodiment, the relative clarity of the second original image relative to the first original image is determined based on the preset weight of each common feature and the corresponding matching degree. Specifically, the step of determining the relative clarity of the second original image relative to the first original image based on the preset weight of each common feature and the corresponding matching degree includes:

[0187] Step S41: based on the preset weight of each common feature and the corresponding matching degree, if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is greater than a preset value, then determining that the second original image is clearer than the first original image;

[0188] Step S42: If the relative clarity of the second original image in the second channel relative to the first original image in the first channel is less than the preset value, determining that the first original image is clearer than the second original image;

[0189] Step S43: If the relative clarity of the second original image in the second channel relative to the first original image in the first channel is equal to the preset value, the clarity between the first original image and the second original image cannot be distinguished.

[0190] That is, in this embodiment, after feature matching is performed on the feature map, two n-dimensional eigenvalues (vectors) Image1Feature1ValueVecter and Image2Feature1ValueVecter are obtained. The corresponding values at the same position in Image1Feature1ValueVecter and Image2Feature1ValueVecter represent the manifestation of the same feature at a certain position in image1 and image2 (i.e., eigenvalues). The larger the eigenvalue, the more obvious the feature manifestation.

[0191] The degree of appearance of some features is positively correlated with clarity, such as the edge information of the image. The more obvious the feature, the clearer the image. Some are negatively correlated, such as the light spot feature of the image. The larger and more obvious the light spot, the blurrier the image.

[0192] Assume that the feature corresponding to the i-th eigenvalue in the eigenvector is positively correlated with the clarity, and the degree of correlation is expressed as W i Indicates. At this time W i It is a positive value. The larger the value, the stronger the correlation. The clarity of image2 relative to image1 can be expressed as follows:

[0193] fvi=(Image2Feature i Value-Image1Feature iValue)*W i

[0194] It can be seen that the more obvious the feature is, the better the Image2Feature i The larger the value, the i The larger it is, the clearer image2 is. On the contrary, if the feature corresponding to the i-th eigenvalue is negatively correlated with the clarity, the correlation degree Wi is a negative value. The smaller the value is, the stronger the negative correlation is. In this case, the feature is more obvious, and Image2Feature i The larger the value, the i The smaller it is, the blurrier image2 is.

[0195] For the result given by the clarity calculation module, the expected result is: when image2 is clearer than image1, fv i >0; when image2 is blurrier than image1, fv i <0; when image2 is as clear as image1 or cannot be compared, fv i =0.

[0196] In order to facilitate the acquisition of parameters such as feature templates in the algorithm and the use of autofocus output results, f V Perform a nonlinear transformation to compress its value range to [0-1], as follows:

[0197]

[0198] FV is the relative clarity of image2 relative to image1. Specifically, when FV > 0.5, image2 is clearer than image1, and the larger the FV, the clearer the image2 is. When FV < 0.5, image2 is blurrier than image1, and the smaller the FV, the blurrier the image2 is. When FV = 0.5, image2 and image1 are equally clear or indistinguishable.

[0199] In this embodiment, the preset value may be 0 or 0.5, and is set according to specific circumstances.

[0200] Step S50: focusing based on the relative clarity.

[0201] In this embodiment, after the relative clarity is obtained, focusing is performed according to the relative clarity, which will not be elaborated in detail.

[0202] The present application provides a camera focusing method, device and computer-readable storage medium. Compared with the prior art in which the camera uses a single-image clarity evaluation algorithm to focus, which causes easy focusing failure, in the present application, a first original image and a second original image captured by the camera are input into a first channel and a second channel respectively for processing, wherein the first original image and the second original image have an intersection in physical space; common features for clarity calculation are extracted from the processed first original image and the processed second original image; the matching degree between the first common feature and the second common feature of the same type is determined; wherein there are multiple common features, the common feature used for clarity calculation in the first original image is the first common feature, and the common feature used for clarity calculation in the second original image is the second common feature; according to the preset weight of each common feature and the corresponding matching degree, the relative clarity of the second original image relative to the first original image is determined; and focusing is performed based on the relative clarity. In the present application, since the clarity evaluation algorithm of a single image is no longer used to calculate the clarity, the common and matching features in the two frames of images (corresponding to the detection of the same physical object) are used to determine the relative clarity of the second original image relative to the first original image. That is, since the same physical object is detected in the two frames of images, the clarity of the two frames of images can be accurately detected. Moreover, since two frames of images are compared instead of a single frame of images, it is considered that the compared images themselves are different, and therefore are less affected by scene changes, which can effectively suppress the interference of dynamic factors, thereby improving the focusing accuracy.

[0203] Furthermore, based on the first embodiment of the present application, another embodiment of the present application is provided. In this embodiment, the step of determining, for common features of the same type, a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number includes:

[0204] Step D1: for common features of the same type, determine a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number, where the matching first feature value is the maximum value of the corresponding block area of the first feature map and the second feature map;

[0205] Step D2: scaling each feature map after each feature extraction and re-extracting the features to obtain multiple groups of feature submaps of different sizes, wherein the smaller the size of the feature map, the lower the resolution of the feature map;

[0206] Step D3: for the common features of the same type, determining a matching second feature value from two feature sub-images of the same resolution and size, wherein the matching second feature value is a maximum value of corresponding block areas of the two feature sub-images of the same resolution and size;

[0207] The step of determining the matching degree between each pair of first feature values as the matching degree between the first common feature and the second common feature of the same type includes:

[0208] In step D4, the matching degree between each pair of first feature values and the matching degree between each pair of second feature values are determined as the matching degree between the first common feature and the second common feature of the same type.

[0209] In this embodiment, it should be noted that when using the maximum eigenvalue for feature matching, the feature map clipping area cannot be too large to avoid feature duplication, resulting in mismatches and affecting subsequent calculation results. However, if the clipping area is too small, excessive feature offsets can lead to matching failures. To address this issue, in this embodiment, a step-by-step scaling and recursive extraction method is used to extract features.

[0210] Specifically, in this embodiment, Figure 10 As shown, each feature map after each feature extraction is scaled and the features are re-extracted to obtain multiple groups of feature submaps of different sizes, wherein the smaller the feature map, the lower the resolution of the feature map. For the same type of common features, the matching second eigenvalue is determined from two feature submaps with the same resolution size. The matching second eigenvalue is based on the maximum value of the corresponding block area of the two feature submaps with the same resolution size, thereby obtaining the maximum and matching second eigenvalue pairs at different levels. The matching degree between each pair of first eigenvalues and the matching degree between each pair of second eigenvalues are determined as the matching degree between the first common feature and the second common feature of the same type.

[0211] In this embodiment, feature matching uses feature maps containing multiple resolutions. This allows only a very small block to be captured at each level to meet the requirements. For example, 3*3 or 5*5 can be adjusted based on actual conditions. In addition, even when the feature offset is large and moves outside the block range of the high-resolution feature map, a successful match can still be achieved due to multi-level matching.

[0212] In the present application, each feature map after each feature extraction is scaled and the features are re-extracted to obtain multiple groups of feature sub-maps of different sizes, and the matching degree between each pair of first eigenvalues and the matching degree between each pair of second eigenvalues are determined as the matching degree between the first common feature and the second common feature of the same type. That is, in the present application, feature matching is performed based on feature maps of multiple levels of different resolutions, that is, feature matching uses feature maps containing multiple resolutions, so that the feature map clipping area is more accurate (the feature map clipping area cannot be too large to avoid feature duplication, resulting in mismatching and affecting subsequent calculation results. However, if the clipping area is too small and the feature offset is too large, matching failure will occur again). In addition, the present application uses multi-level feature maps, which can take into account both high-resolution and low-resolution features, making the algorithm more adaptable and the focus more accurate.

[0213] Furthermore, based on the first embodiment of the present application, another embodiment of the present application is provided. In this embodiment, the preset weights of the common features are determined when the minimum value of a preset loss function is obtained based on training data with preset classification labels. When obtaining the minimum value of the preset loss function, the calculation is based on a gradient descent algorithm, and the matching degree is determined based on the feature difference between the first common feature and the second common feature.

[0214] In this embodiment, the training data is first labeled with types to obtain training data with preset classification labels, and then based on the constructed training data, the loss function is calculated based on the gradient descent algorithm to obtain the preset weights of the common features.

[0215] In this embodiment, when annotating data, it is only necessary to record the relatively clear types in the sequence images (such as Figure 13 As shown, there are 6 images in an image sequence, among which the clearest image is numbered 5. Then, two images are randomly selected on the side smaller than 5, numbered 2 and 4 respectively. Since number 4 is closer to 5, image 4 shifted to the left is clearer than image 2. Therefore, we only need to record the relatively clear types in the sequence images), that is, this application treats the regression problem as a classification problem.

[0216] Specifically, if the preset value is 0.5, during model training, if the input image becomes sharp, it is labeled as 0.5+δ. Correspondingly, if the image becomes blurry, it is labeled as 0.5-δ. Where δ is a small positive number used to ensure that the algorithm can learn the correct classification of the input image.

[0217] Specifically, in this embodiment, the preset weights of the common features are obtained through training, and the training process is as follows:

[0218] Assume that the input images are image1 and image2 of two channels, and the model parameters are W (including all undetermined parameters, such as feature template element values, clarity correlation parameters, etc.). The algorithm output FV is a function of I and W, as follows:

[0219] FV=f(I,V)

[0220] Assume that the initial value of parameter W is random. For any input I, the expected output is FV, and the actual output is The deviation function can be constructed: By minimizing the value of L(W), we can find the appropriate parameter W. The loss function L(W) is differentiable with respect to the parameter W, so W can be obtained by the gradient descent algorithm as follows:

[0221]

[0222] Where η is the learning rate, It is the partial derivative of the loss function with respect to the parameters, also known as the gradient. In this embodiment, the direction of the parameter gradient is the direction in which the function value increases, and its opposite direction is the direction in which the function decreases. By simply adjusting the parameters in the opposite direction of the gradient, the function value can be reduced, thereby achieving the purpose of optimizing the parameters. Among them, the learning rate η is used to control the step size of the parameter adjustment.

[0223] In this embodiment, the problem of data being unable to be labeled is solved by converting the regression problem into a classification problem.

[0224] Further, refer to Figure 2 Based on the above embodiments of the present application, another embodiment of the present application is provided, including:

[0225] Step G10: selecting a first image and a second image, wherein the second image comprises partially identical image content and partially different image content, wherein the partially different image content is in response to dynamic interference imposed by a dynamic object;

[0226] As an example, the camera selects the first image and the second image by itself, wherein the second image contains partially identical image content and partially different image content, and the partially different image content is in response to the dynamic interference imposed by the dynamic object.

[0227] As an example, a dynamic object may be an object that causes a camera to shake, including but not limited to factors such as wind and hands.

[0228] Step G20, inputting the first image via the first channel and inputting the second image via the second channel;

[0229] Step G30, performing grayscale processing and normalization processing on the input first image and the second image;

[0230] Performing feature extraction on the normalized first image and the second image, comprising:

[0231] Step G40: performing matching on the normalized first image and the second image in response to a plurality of preset feature templates, and determining a plurality of first feature maps and a plurality of second feature maps that match the plurality of preset feature templates, wherein any pixel value in the first feature map is a matrix product value of the preset feature template and the normalized first image, and any pixel value in the second feature map is a matrix product value of the preset feature template and the normalized second image;

[0232] Step G50, determining a clarity value of the second image relative to the first image, wherein the clarity value is determined by a difference weight between the second feature map and the associated first feature map;

[0233] Step G60: determining the focus curve of the camera according to the clarity value.

[0234] The first image and the second image are defined as images in an image sequence generated during the process of moving the focusing lens group from the farthest end to the near end.

[0235] Among them, any pixel value in the first feature map W (i,j) Represents the element value of the feature template at (i, j), I (x-1+i,y-1+j) represents the pixel value of the image of the same size as the feature template that is intercepted from the normalized first image, such as Figure 9 shown.

[0236] The clarity value of the second image relative to the first image is calculated by taking the difference between each pair of matching feature values, multiplying them by different weights according to the feature type and matching situation, and taking the average of the accumulated sums as the final result of measuring relative clarity to suppress the interference of dynamic factors, where:

[0237] The clarity value of the second image relative to the first image

[0238]

[0239] Among them, Image2Feature i Value and Image1Feature i Value is the matching feature value pair of the second image and the first image corresponding to the i-th matching feature, and

[0240] Image2Feature i Value-Image1Featurei Value is the matching status of the i-th matching feature, w i is the influence weight of the i-th matching feature on the clarity.

[0241] Reference Figure 3 , Figure 3 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0242] like Figure 3 As shown, the camera focusing device may include: a processor 1001 , a memory 1005 , and a communication bus 1002 . The communication bus 1002 is used to implement connection and communication between the processor 1001 and the memory 1005 .

[0243] Optionally, the camera focusing device may further include a user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, a WiFi module, and the like. The user interface may include a display and an input submodule such as a keyboard. The optional user interface may also include a standard wired interface and a wireless interface. The network interface may include a standard wired interface and a wireless interface (such as a WiFi interface).

[0244] Those skilled in the art will understand that Figure 3 The camera focusing device structure shown in the figure does not constitute a limitation to the camera focusing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0245] like Figure 3 As shown, memory 1005, a computer-readable storage medium, may include an operating system, a network communication module, and a camera focus program. The operating system is a program that manages and controls the hardware and software resources of the camera focus device and supports the operation of the camera focus program and other software and / or programs. The network communication module is used to enable communication between the various components within memory 1005, as well as communication with other hardware and software in the camera focus system.

[0246] exist Figure 3 In the camera focusing device shown, the processor 1001 is used to execute the camera focusing program stored in the memory 1005 to implement the steps of any one of the above-mentioned camera focusing methods.

[0247] The specific implementation of the camera focusing device of the present application is basically the same as the embodiments of the above-mentioned camera focusing method, and will not be repeated here.

[0248] The present application also provides a camera focusing device, the device comprising:

[0249] An input module, configured to input a first original image and a second original image captured by a camera into a first channel and a second channel respectively for processing, wherein the first original image and the second original image have an intersection in physical space;

[0250] a feature extraction module, configured to extract common features for clarity calculation from the processed first original image and the processed second original image;

[0251] a feature matching module, configured to determine a degree of matching between a first common feature and a second common feature of the same type; wherein the common features are multiple, the common feature used for clarity calculation in the first original image is the first common feature, and the common feature used for clarity calculation in the second original image is the second common feature;

[0252] a clarity calculation module, configured to determine a relative clarity of the second original image relative to the first original image based on a preset weight of each common feature and the corresponding matching degree;

[0253] A result output module is used to focus based on the relative clarity.

[0254] In a possible implementation of the present application, the input module is configured to: input the first original image into the first channel, and input the second original image into the second channel; the feature extraction module includes: an extraction unit configured to extract common features for clarity calculation in the first image and the second image;

[0255] And / or the extraction unit is used to implement: determining a feature template, wherein the element value of each feature template is obtained based on preset annotation data and learned by a preset algorithm model, and there are multiple feature templates; performing feature extraction on each image according to the preset feature template to obtain a set of first feature maps of the first image and a set of second feature maps of the second image, wherein the first feature maps and the second feature maps are numbered in the same order, and each feature map is composed of feature values of multiple common features, and the feature values are calculated values corresponding to the area in each image that is convolved with the feature template; the feature matching module is used to determine, for the same type of common features, the matching first feature value from the first feature map and the second feature map corresponding to the same sequence number, the matching first feature value being the maximum value of the corresponding block area of the first feature map and the second feature map; and determining the matching degree between each pair of first feature values as the matching degree between the first common feature and the second common feature of the same type;

[0256] And / or the feature matching module is configured to: determine, for a common feature of the same type, a matching first feature value from a first feature map and a second feature map corresponding to the same sequence number, wherein the matching first feature value is a maximum value of corresponding block areas in the first feature map and the second feature map, wherein, when the common feature of the same type is associated with a moving object, a size of the block area is associated with a moving speed of the moving object;

[0257] And / or the clarity calculation module includes: a first determining unit, configured to, based on a preset weight of each common feature and the corresponding matching degree, determine that the second original image in the second channel is clearer than the first original image in the first channel if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is greater than a preset value; a second determining unit, configured to determine that the first original image is clearer than the second original image if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is less than the preset value; and a third determining unit, configured to determine that the clarity between the first original image and the second original image is indistinguishable if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is equal to the preset value;

[0258] And / or the preset weights of the common features are determined when obtaining the minimum value of a preset loss function based on training data with preset classification labels. When obtaining the minimum value of the preset loss function, the calculation is based on a gradient descent algorithm, and the matching degree is determined based on the feature difference between the first common feature and the second common feature.

[0259] The specific implementation of the camera focusing device of the present application is basically the same as the embodiments of the above-mentioned camera focusing method, and will not be repeated here.

[0260] The present application also provides a camera focusing device, which is used to achieve:

[0261] Focusing the camera, selecting a first image and a second image, wherein the second image includes a portion of identical image content and a portion of different image content, wherein the portion of different image content is in response to a dynamic disturbance imposed by a dynamic object;

[0262] Inputting the first image via a first channel and inputting the second image via a second channel;

[0263] performing grayscale processing and normalization processing on the input first image and the second image;

[0264] Performing feature extraction on the normalized first image and the second image, comprising:

[0265] In response to a plurality of preset feature templates, matching is performed on the normalized first image and the second image to determine a plurality of first feature maps and a plurality of second feature maps that match the plurality of preset feature templates, wherein any pixel value in the first feature map is a matrix product value of the preset feature template and the normalized first image, and any pixel value in the second feature map is a matrix product value of the preset feature template and the normalized second image;

[0266] determining a clarity value of the second image relative to the first image, the clarity value being weighted by a difference between the second feature map and the associated first feature map;

[0267] A focus curve of the camera is determined according to the clarity value.

[0268] In a possible implementation manner of the present application, the device is used to implement: the first image and the second image are defined as images in an image sequence generated during the process of moving the focusing lens group from the farthest end to the near end.

[0269] In a possible implementation manner of the present application, the device is used to implement:

[0270] Any pixel value in the first feature map W (i,j) Represents the element value of the feature template at (i, j), I (x-1+i,y-1+j) Represents the pixel value of a picture of the same size as the feature template that is captured from the normalized first image.

[0271] In one possible implementation of the present application, the apparatus is configured to implement: a clarity value of the second image relative to the first image = subtracting each pair of matching feature values, multiplying them by different weights according to feature type and matching status, and taking the average of the accumulated sums as the final result of measuring relative clarity to suppress interference from dynamic factors, wherein:

[0272] The clarity value of the second image relative to the first image

[0273] Among them, Image2Feature i Value and Image1Feature i Value is the matching feature value pair of the second image and the first image corresponding to the i-th matching feature, and Image2Feature i Value-Image1Feature i Value is the matching status of the i-th matching feature, w iis the influence weight of the i-th matching feature on the clarity.

[0274] The specific implementation of the camera focusing device of the present application is basically the same as the embodiments of the above-mentioned camera focusing method, and will not be repeated here.

[0275] An embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of any of the above-mentioned camera focusing methods.

[0276] The specific implementation of the computer-readable storage medium of the present application is basically the same as the embodiments of the above-mentioned camera focusing method, and will not be repeated here.

[0277] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned camera focusing method when executed by a processor.

[0278] The specific implementation of the computer program product of the present application is basically the same as the embodiments of the above-mentioned camera focusing method, and will not be repeated here.

[0279] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0280] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0281] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus hardware platform, or by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM) and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0282] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A camera focusing method, characterized in that: The method comprises: Inputting a first original image and a second original image captured by a camera into a first channel and a second channel respectively for processing, wherein the first original image and the second original image have an intersection in physical space; extracting common features for clarity calculation from the processed first original image and the processed second original image; Determining a degree of matching between a first common feature and a second common feature of the same type; wherein the common features are multiple, the common feature used for clarity calculation in the first original image is the first common feature, and the common feature used for clarity calculation in the second original image is the second common feature; determining a relative clarity of the second original image relative to the first original image based on a preset weight of each common feature and the corresponding matching degree; Focusing is performed based on the relative sharpness.

2. The camera focusing method according to claim 1, wherein: The step of inputting the first original image and the second original image captured by the camera into the first channel and the second channel respectively for processing comprises: Input the first original image into the first channel, and input the second original image into the second channel; The common features used for clarity calculation in the first original image after the extraction process and the second original image after the processing include: Gray-scaling, normalizing, and scaling the first original image and the second original image are performed in corresponding channels, and scaling partial regions are intercepted to obtain first and second images; Extracting common features for clarity calculation from the first image and the second image.

3. The camera focusing method according to claim 2, wherein: The step of extracting common features for clarity calculation from the first image and the second image includes: Determining feature templates, wherein element values of each feature template are obtained based on preset annotation data and learned through a preset algorithm model, and there are multiple feature templates; Performing feature extraction on each image according to the feature template to obtain a set of first feature maps for the first image and a set of second feature maps for the second image, wherein the first feature maps and the second feature maps are numbered in the same order, each feature map is composed of feature values of a plurality of common features, and the feature values are calculated values corresponding to regions in each image that undergo convolution calculation with the feature template; The step of determining the degree of matching between the first common feature and the second common feature of the same type comprises: For common features of the same type, determine a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number. The matching first feature value is the maximum value of the corresponding block area of the first feature map and the second feature map; The matching degree between each pair of first feature values is determined as the matching degree between the first common feature and the second common feature of the same type.

4. The camera focusing method according to claim 3, wherein: The step of determining the matching first feature value from the first feature map and the second feature map corresponding to the same sequence number for the common features of the same type includes: For common features of the same type, determining a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number; Each feature map after each feature extraction is scaled and features are re-extracted to obtain multiple sets of feature submaps of different sizes, where the smaller the size of the feature map, the lower the resolution of the feature map; For common features of the same type, a matching second eigenvalue is determined from two feature sub-images with the same resolution. The matching second eigenvalue is based on the maximum value of the corresponding block areas of the two feature sub-images with the same resolution. The step of determining the matching degree between each pair of first feature values as the matching degree between the first common feature and the second common feature of the same type includes: The matching degree between each pair of first eigenvalues and the matching degree between each pair of second eigenvalues are both determined as the matching degree between the first common feature and the second common feature of the same type.

5. The camera focusing method according to claim 3, wherein: The step of determining, for the common features of the same type, a matching first feature value from the first feature map and the second feature map corresponding to the same sequence number, wherein the matching first feature value is a maximum value of the corresponding block areas of the first feature map and the second feature map, includes: For common features of the same type, a matching first feature value is determined from the first feature map and the second feature map corresponding to the same sequence number, and the matching first feature value is the maximum value of the block area corresponding to the first feature map and the second feature map, wherein, when the common features of the same type are associated with a moving object, the size of the block area is correlated with the moving speed of the moving object.

6. The camera focusing method according to claim 1, wherein: The step of determining the relative clarity of the second original image relative to the first original image based on the preset weight of each common feature and the corresponding matching degree includes: According to the preset weight of each common feature and the corresponding matching degree, if the relative clarity of the second original image in the second channel relative to the first original image in the first channel is greater than a preset value, then determining that the second original image is clearer than the first original image; If the relative clarity of the second original image in the second channel relative to the first original image in the first channel is less than the preset value, determining that the first original image is clearer than the second original image; If the relative clarity of the second original image in the second channel relative to the first original image in the first channel is equal to the preset value, the clarity between the first original image and the second original image cannot be distinguished.

7. A camera focusing method, comprising: selecting a first image and a second image, wherein the second image comprises a portion of identical image content and a portion of different image content, wherein the portion of different image content is in response to a dynamic disturbance imposed by a dynamic object, and the first image and the second image are defined as images in an image sequence generated during the process of moving the focusing lens group from the farthest end to the near end; Inputting the first image via a first channel and inputting the second image via a second channel; performing grayscale processing and normalization processing on the input first image and the second image; Performing feature extraction on the normalized first image and the second image, comprising: In response to a plurality of preset feature templates, matching is performed on the normalized first image and the second image to determine a plurality of first feature maps and a plurality of second feature maps that match the plurality of preset feature templates, wherein any pixel value in the first feature map is a matrix product value of the preset feature template and the normalized first image, and any pixel value in the second feature map is a matrix product value of the preset feature template and the normalized second image; determining a clarity value of the second image relative to the first image, the clarity value being determined by a difference weight between the second feature map and the associated first feature map; Based on the clarity value, a focus curve of the camera is determined.

8. The method according to claim 7, wherein: Any pixel value in the first feature map W ij represents the element value of the feature template at (i, j), and I(x-1+i, y-1+j) represents the pixel value in a picture of the same size as the feature template that is cut out from the normalized first image.

9. The method according to claim 7, wherein: The clarity value of the second image relative to the first image is calculated by taking the difference between each pair of matching feature values, multiplying them by different weights according to the feature type and matching situation, and taking the average of the accumulated sums as the final result of measuring relative clarity to suppress the interference of dynamic factors, where: The clarity value of the second image relative to the first image Among them, Image2Feature i Value and Image1Feature i Value is the matching feature value pair of the second image and the first image corresponding to the i-th matching feature, and Image2Feature i Value-Image1Feature i Value is the matching status of the i-th matching feature, w i is the influence weight of the i-th matching feature on the clarity.

Images