Image processing method, device, apparatus and storage medium

Abstract

The application provides an image processing method, device and equipment and a storage medium. In the focusing process, a first prediction result of a current image and a second prediction result of a previous frame image are output by a trained convolutional neural network, and an incremental data vector is constructed based on the first prediction result and the second prediction result. The incremental data vector is input into the trained recurrent neural network, so that the recurrent neural network can combine the changes of multiple frames of images in time sequence in the process of determining a predicted focusing distance, the focusing accuracy is improved, the incremental data vector is input into the recurrent neural network in real time after each frame of image is captured, the predicted focusing distance is dynamically updated, the shooting is dynamically terminated in combination with credibility and the number of shooting times, and complete multiple frames of images are not needed, so that redundant shooting is reduced.

Description

Technical Field

[0001] This application belongs to the field of automotive technology, specifically relating to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Electron microscopes, as a key tool in modern technology, play an indispensable role in fields such as materials science, semiconductor technology, and biology. However, in the autofocus process, the frequent projection of electron beams onto the target area to capture images poses a potential risk of damage to sensitive materials such as semiconductor products, as these materials often have strict limitations on the number of electron beam irradiations.

[0003] Furthermore, in relevant scenarios, there are also relatively high requirements for focusing accuracy and focusing efficiency. Summary of the Invention

[0004] This application proposes an image processing method, apparatus, device, and storage medium that can solve the technical problems in related technologies, such as redundant shooting times, low focusing accuracy and efficiency during focusing, and difficulty in efficiently capturing and analyzing related fault information.

[0005] The first aspect of this application provides an image processing method, including:

[0006] While focusing and acquiring the current image captured by the imaging device, the following iterative process is performed:

[0007] The current image is input into the trained convolutional neural network to obtain a first prediction result, and the previous frame image is input into the trained convolutional neural network to obtain a second prediction result;

[0008] An incremental data vector is constructed based on the first prediction result and the second prediction result;

[0009] The incremental data vector is input into the trained recurrent neural network to obtain a third prediction result, which includes: predicted focus distance and first confidence level.

[0010] If the first focusing completion condition is met, the iteration process is completed, and the imaging device is controlled to capture the target image based on the predicted focusing distance. The first focusing completion condition is that the first confidence level is greater than the preset confidence level threshold.

[0011] An embodiment of the second aspect of this application provides an image processing apparatus, comprising:

[0012] The execution module is used to perform the following iterative process while the focusing process is in progress and the current image captured by the imaging device is being acquired:

[0013] The acquisition module is used to acquire the first prediction result obtained by inputting the current image into the trained convolutional neural network and the second prediction result obtained by inputting the previous frame image into the trained convolutional neural network;

[0014] A construction module is used to construct an incremental data vector based on the first prediction result and the second prediction result;

[0015] The input module is used to input the incremental data vector into the trained recurrent neural network to obtain a third prediction result, the third prediction result including: predicted focus distance and first confidence level;

[0016] The control module is used to complete the iterative process if the first focusing completion condition is met, and to control the imaging device to capture the target image based on the predicted focusing distance. The first focusing completion condition is that the first confidence level is greater than a preset confidence level threshold.

[0017] An embodiment of the third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0018] An embodiment of the fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the first aspect above.

[0019] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0020] This application discloses an image processing method, apparatus, device, and storage medium, the method comprising:

[0021] This application proposes an image processing method, apparatus, device, and storage medium. In the embodiments of this application, during the focusing process, a trained convolutional neural network outputs a first prediction result for the current image and a second prediction result for the previous frame image, and constructs an incremental data vector based on the first and second prediction results. This incremental data vector is then input into the trained recurrent neural network, enabling the recurrent neural network to improve focusing accuracy by incorporating the temporal changes of multiple frames in determining the predicted focus distance. Furthermore, after each frame is captured, an incremental data vector is input into the recurrent neural network in real time to dynamically update the predicted focus distance. The system dynamically terminates shooting based on confidence level and the number of shots, eliminating the need to wait for complete multi-frame images and reducing redundant shooting.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0024] Figure 1 A flowchart of an image processing method provided in an embodiment of this application is shown;

[0025] Figure 2 A schematic diagram of an eight-3×3 convolution operator provided in an embodiment of this application is shown;

[0026] Figure 3 A flowchart of an image processing method provided in an embodiment of this application is shown;

[0027] Figure 4 This invention provides a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present application.

[0028] Figure 5 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0029] Figure 6 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0030] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0031] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0032] The image processing method of this application can be executed by a computing device, which can be a server, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, such as a mobile phone, tablet computer, game console, portable computer, desktop computer, advertising machine, all-in-one machine, etc. This application does not limit the type or number of computing devices.

[0033] The following description, in conjunction with the accompanying drawings, illustrates an image processing method according to an embodiment of this application. This embodiment uses a vehicle-mounted terminal as the execution subject to illustrate the image processing method.

[0034] See Figure 1 The method specifically includes the following steps:

[0035] While focusing and acquiring the current image captured by the imaging device, the following iterative process is performed:

[0036] S101. Obtain the first prediction result obtained by inputting the current image into the trained convolutional neural network and the second prediction result obtained by inputting the previous frame image into the trained convolutional neural network.

[0037] S102. Construct an incremental data vector based on the first and second prediction results.

[0038] S103. Input the incremental data vector into the trained recurrent neural network to obtain the third prediction result.

[0039] The third prediction result includes: predicted focus distance and first confidence level.

[0040] S104. If the first focusing completion condition is met, the iterative process is completed, and the imaging device is controlled to capture the target image based on the predicted focusing distance.

[0041] The first focus completion condition is that the first confidence level is greater than the preset confidence level threshold.

[0042] The imaging equipment can be an electron microscope, a camera, etc.

[0043] The focusing process can be triggered by receiving a trigger command or detecting a significant change in the current scene.

[0044] In some embodiments, only the actual pattern obtained on the imaging plane (sensor / screen) fully contains all aberrations, defocus, image shift, and noise information of the entire "current optical system + sample / scene" chain. That is, focusing can only be completed after obtaining the image captured by the imaging device. Therefore, during the focusing process of the imaging device, some images are captured for focusing processing.

[0045] Once the current image captured by the imaging device is obtained, it can be input into a trained convolutional neural network (CNN) to obtain the first prediction result for the current image.

[0046] The current image can be any frame captured by the imaging device during the focusing process.

[0047] After obtaining the first prediction result of the current image, in order to obtain the temporal changes between multiple frames, the second prediction result of the previous frame image can be obtained by inputting the previous frame image into the trained convolutional neural network.

[0048] Furthermore, an incremental data vector is constructed based on the first and second prediction results.

[0049] In some embodiments, the focus during focusing is mainly on changes in image sharpness and background. Therefore, the first prediction result and the second prediction result may include the sharpness and image features corresponding to the two frames respectively. Thus, the temporal correlation parameters can be determined based on the sharpness and image features corresponding to the two frames respectively, so as to reflect the changes in sharpness and background of the two frames.

[0050] In addition, the first prediction result also includes the first predicted focus offset.

[0051] In some embodiments, an incremental data vector is constructed based on the first prediction result and the temporal correlation parameters; the incremental data vector is input into a trained recurrent neural network (RNN), so that the trained recurrent neural network can combine the sharpness change, background change and first predicted focus offset of the two frames to output the predicted focus distance and the first confidence level.

[0052] The predicted focus distance is the optimal focus distance output by the trained recurrent neural network, and the first confidence level is used to reflect the reliability of the current optimal focus distance prediction value.

[0053] After obtaining the first confidence level, it is first determined whether the first confidence level is greater than the preset confidence level threshold. If it is greater than the preset confidence level threshold, it means that the current output predicted focus distance is relatively reliable, and the shooting can be carried out according to the predicted focus distance. Furthermore, the imaging device can be controlled to move to the shooting position corresponding to the predicted focus distance to take a picture in order to obtain the target image. The preset confidence level threshold can be flexibly set based on the actual situation, such as 0.95.

[0054] This application proposes an image processing method. In the embodiments of this application, during the focusing process, a trained convolutional neural network outputs a first prediction result of the current image and a second prediction result of the previous frame image, and constructs an incremental data vector based on the first and second prediction results. The incremental data vector is then input into a trained recurrent neural network, which enables the recurrent neural network to combine the temporal changes of multiple frames in the process of determining the predicted focus distance, thereby improving the focusing accuracy. Furthermore, after each frame image is captured, the incremental data vector is input into the recurrent neural network in real time to dynamically update the predicted focus distance. The shooting is dynamically terminated based on the confidence level and the number of shots, without waiting for complete multiple frames of images, thus reducing redundant shooting.

[0055] In some embodiments, the method further includes: if the second focusing completion condition is met, then completing the iterative process, determining the target confidence level from multiple first confidence levels output by the trained recurrent neural network during the focusing process, wherein the second focusing completion condition is that the number of shots corresponding to the current image reaches a preset number of shots threshold; obtaining the target predicted focusing distance corresponding to the target confidence level, and controlling the imaging device to capture the target image based on the target predicted focusing distance.

[0056] Understandably, in order to protect the imaging equipment, the number of shots taken during the focusing process of the imaging equipment should not exceed a preset threshold. The preset threshold can be flexibly set based on the actual situation of the imaging equipment, such as 5 shots.

[0057] If the first confidence level is determined to be no greater than the preset confidence level threshold, the number of times the current image is captured in this focusing process can be determined. If the number of captures reaches the preset capture number threshold, multiple first confidence levels output by the trained RNN in multiple iterations during this focusing process can be obtained. The target confidence level can be determined from multiple first confidence levels. For example, the largest first confidence level can be determined as the target confidence level, and the predicted focusing distance corresponding to the target confidence level can be obtained. This predicted focusing distance can be determined as the target predicted focusing distance. Furthermore, the imaging device can be controlled to move to the shooting position corresponding to the target predicted focusing distance to capture the image and obtain the target image.

[0058] In some embodiments, the method further includes: if the first focus completion condition or the second focus completion condition is not met, obtaining the first predicted focus offset of the first prediction result and the first focus distance corresponding to the current image; determining the deviation direction based on the predicted focus distance and the first focus distance; determining the second focus distance based on the first predicted focus offset and the first focus distance; obtaining the next frame image captured by the imaging device based on the second focus distance; inputting the next frame image and the deviation direction into the trained convolutional neural network, and continuing to iterate until the first focus completion condition or the second focus completion condition is met.

[0059] In some embodiments, if the first confidence level of the trained RNN output is not greater than the confidence level threshold, and the number of times the current image has been captured has not reached the preset capture number threshold, then it is necessary to capture the next frame image to proceed with the next iteration.

[0060] Specifically, the first predicted focus offset and the first focus distance corresponding to the current image can be obtained from the first prediction result. The first focus distance is the focus distance corresponding to the shooting position of the current image. The first predicted focus offset is the absolute value and direction of the difference between the optimal focus distance predicted by the CNN and the first focus distance. For example, if the first predicted focus offset is +0.08 mm, and + represents moving forward, then +0.08 mm means moving forward 0.08 mm from the shooting position of the current image.

[0061] The second focus distance can be determined based on the focus distance offset and the first focus distance. The second focus distance is the image corresponding to the shooting position of the next frame image.

[0062] Furthermore, the deviation direction is determined based on the predicted focusing distance and the first focusing distance. Assuming the predicted focusing distance is 0.94 mm and the first focusing distance is 0.88 mm, the predicted focusing distance is greater than the first focusing distance, and the deviation direction is towards the telephoto.

[0063] The next frame image and the deviation direction are input into the trained CNN. The trained CNN adjusts the image analysis focus of the next frame according to the deviation direction: if the deviation direction is "far focus", the weight of the representation data of the far focus area of ​​the image (such as the edge of the screen) is increased by 20% during the model processing, and the focus-related output is calculated based on the data of this area first, reducing the interference of invalid data.

[0064] The trained CNN outputs the prediction result of the next frame image, and constructs corresponding temporal correlation parameters based on the first prediction result of the current image and the prediction result of the next frame image. Based on the prediction result of the next frame image and the temporal correlation parameters, a new incremental data vector is constructed and input into the trained RNN. The predicted focus distance and the first confidence level output by the RNN are updated until the first confidence level corresponding to a certain frame image meets the first focus completion condition or the number of shots for a certain frame image meets the preset number of shots requirement, so as to obtain the target image.

[0065] In some embodiments, the first prediction result further includes a first sharpness value and a first image feature, and the second prediction result includes a second sharpness value and a second image feature. Determining temporal correlation parameters based on the first and second prediction results includes: calculating a feature matching degree based on the first and second image features; obtaining a third focus distance of the previous frame image; calculating a focus distance difference based on the first and third focus distances; calculating a sharpness difference based on the first and second sharpness values; determining a sharpness change slope based on the sharpness difference and the focus distance difference; combining the feature matching degree and the sharpness change slope to obtain temporal correlation parameters; and constructing an incremental data vector by combining the first predicted focus offset and the temporal correlation parameters.

[0066] In some embodiments, the CNN model can output not only the predicted focus offset, but also sharpness and image features.

[0067] Feature matching degree can be the cosine similarity of image features between the current image and the previous frame, reflecting the degree of similarity between the two frames at the feature level. Feature matching degree is used to reflect scene stability: when the feature matching degree is close to 1, it indicates high similarity between the previous and next frames, with no significant change in the scene, and the RNN will appropriately increase the weight of historical incremental data when calculating the predicted focus distance; when the feature matching degree is much lower than 1, it indicates low similarity between the previous and next frames, suggesting a sudden change in the scene, and the RNN will appropriately reduce the weight of historical incremental data when calculating the predicted focus distance.

[0068] Further, the slope of the sharpness change reflects the trend of sharpness change with the adjustment of the focus distance. The focus trend can be predicted based on the slope. For example, if the slope is positive and the value is large, it means that the current focus direction is correct, and the RNN can accelerate the prediction of the optimal focus distance for the current image in that direction; if the slope is negative, it means that the focus direction is incorrect, and the RNN can adjust the prediction in reverse in time to reduce invalid iterations.

[0069] In some embodiments, a sharpness difference can be calculated based on a first sharpness value and a second sharpness value; a focus distance difference can be calculated based on a first focus distance and a third focus distance; and a sharpness change slope can be determined based on the sharpness difference and the focus distance difference.

[0070] In some embodiments, the sharpness difference is a first sharpness value minus a second sharpness value, the focus distance difference is a first focus distance minus a third focus distance, and the slope of the sharpness change is the ratio of the sharpness difference to the focus distance difference.

[0071] In some embodiments, the first prediction result also includes a first predicted focus offset, a first sharpness value, and a second confidence level corresponding to the first image feature set. An incremental data vector can also be constructed by combining time-series correlation parameters, the first predicted focus offset, and the second confidence level.

[0072] The feature matching degree and the slope of sharpness change are combined to obtain the temporal correlation parameters. The incremental data vector is constructed by combining the first predicted focus offset and then used as the temporal correlation parameters to input the trained RNN.

[0073] In some embodiments, the process of determining the first sharpness value includes: dividing the current image into multiple regions; calculating the grayscale threshold of each region and the average grayscale threshold of all regions; determining regions with grayscale thresholds greater than the average grayscale threshold as target regions; calculating the grayscale variance of each target region; if the grayscale variance of the first target region is less than the grayscale threshold of the first target region, then setting the grayscale gradient of the first target region to 0, and the first target region is any target region among the target regions; if the grayscale variance of the first target region is greater than or equal to the grayscale threshold of the first target region, then calculating the grayscale gradient of the first target region; and determining the first sharpness value based on the grayscale gradient of each target region.

[0074] In some embodiments, the initial image sharpness value can be calculated using the Laplacian operator and the local maximum gradient image sharpness evaluation function. The calculation process is as follows:

[0075] For the current image, it can be divided into multiple regions. For example, if the current image size is M×N, the image can be divided into regions of size m×n (for example, if the image size is 3088×2064, then the image can be divided into 64 regions of size 8×8).

[0076] Convolution operations can be performed on each region according to equation (1) to highlight high-frequency information in the image, and the gray-level threshold of each region and the mean of the gray-level thresholds of all regions can be calculated:

[0077] (1)

[0078] Among them, u 1ij Let f(x,y) be the average gray value of all elements contained in each region, M be the number of rows in the current image, N be the number of columns in the current image, m be the number of rows in each region, n be the number of columns in each region, f(x,y) be any region among the multiple regions, x be the number of rows in the region, y be the number of columns in the region, i be the number of rows of a certain pixel, and j be the number of columns of the pixel.

[0079] Based on the calculation results of the grayscale threshold and the average grayscale threshold of all regions, the regions with grayscale thresholds greater than the average grayscale threshold are identified, and these regions are determined as target regions, i.e., the regions for subsequent image sharpness calculation.

[0080] Traverse all pixels within each target region. In this embodiment, the target region is... For all pixels within a 3×3 neighborhood centered on the center, calculate the pixel value using the following formula. It is the grayscale variance of all pixels within a 3×3 neighborhood centered on the center.

[0081] (2)

[0082] in, The grayscale mean of each target region is the average grayscale value of all elements in each 3×3 neighborhood. The gray-level variance for each target region is the mean gray-level value of all elements in each 3×3 neighborhood.

[0083] Compare the obtained gray variance with the gray threshold of the corresponding region. If the gray variance of a certain target region is less than the gray threshold of all target regions, then the target region is regarded as background or noise and its gray gradient is set to zero. If the gray variance of a certain target region is greater than or equal to the gray threshold of all target regions, then the 8 pixels near the center point of the image are processed according to Equation (3) and Equation (4) to obtain the maximum value as the expected gradient.

[0084] (3)

[0085] (4)

[0086] in, Let the grayscale gradient of any target region be . To obtain the maximum value, For convolution operations, to The eight 3×3 convolution operators used to calculate the maximum gradient are expressed as follows: Figure 2 As shown.

[0087] Furthermore, the sharpness value is calculated using equation (5):

[0088] (5)

[0089] Here, Q represents the sharpness value, and Q is the number of target regions that yield the maximum grayscale gradient. The first sharpness value is obtained by summing the grayscale gradients of each target region.

[0090] In some embodiments, the second prediction result further includes a second predicted focus offset. The trained recurrent neural network is equipped with a gated recurrent unit, which includes a reset gate and an update gate. The incremental data vector is input into the trained recurrent neural network to obtain a third prediction result, including: determining a first output value of the reset gate based on the feature matching degree; determining a second output value of the update gate based on the sharpness change slope; determining a candidate hidden state based on the first output value, the first predicted focus offset, and the second predicted focus offset; determining a final hidden state based on the candidate hidden state and the second output value; determining a predicted focus distance based on the final hidden state and the sharpness change slope; calculating a first confidence level of the predicted focus distance; and obtaining a third prediction result based on the predicted focus distance and the first confidence level.

[0091] In some embodiments, the RNN includes a gated recurrent unit (GRU), which includes an update gate with a reset gate.

[0092] After receiving the incremental data vector, the RNN can obtain the feature matching degree and the slope of the sharpness change. Scene stability is judged based on the feature matching degree: when the feature matching degree is close to 1, it indicates that there is no significant change in the scene between consecutive frames; when the feature matching degree is much lower than 1, it indicates a sudden change in the scene. The reset gate controls whether historical hidden states are forgotten. The larger the output value of the control gate, the more correlated the RNN output is with the historical hidden states; conversely, the smaller the output value of the control gate, the less correlated the RNN output is with the historical hidden states. Therefore, the output coefficient of the control gate is determined by the feature matching degree.

[0093] Specifically, when the feature matching degree is close to 1, it indicates that there is no significant change in the scene between the previous and next frames. It is necessary to combine the historical hidden state to determine the RNN output, and the output coefficient of the control gate increases. If the feature matching degree is much lower than 1, it indicates that the scene has changed abruptly, and the historical hidden state needs to be forgotten. Therefore, the output coefficient of the control gate decreases.

[0094] The historical hidden state can be updated based on the output coefficients of the control gate. Since the output of the RNN for each frame image must be combined with the data of the previous frame image, the focus offset of the previous frame image includes the focus offset of each iteration process in this focusing process. Therefore, the historical hidden state at this time can be the focus offset of the previous frame image, that is, the second predicted focus offset.

[0095] Correspondingly, the candidate hidden state can be determined based on the first predicted focus offset, the second predicted focus offset, and the reset gate output coefficient. The candidate hidden state is a fusion of the current input and the historical state after reset, where the current input is the first predicted focus offset.

[0096] Correspondingly, the candidate hidden state can be obtained through equation (6):

[0097] (6)

[0098] in, In the candidate hidden state, The input at the current moment is the first predicted focus offset. The output of the reset gate corresponding to the current input is the first output value. This is the weight matrix. This is the input from the previous moment, i.e., the second predicted focus offset. The bias vector, weight matrix, and bias vector can be flexibly set based on the actual situation.

[0099] After determining the candidate hidden state, the candidate hidden state can be adjusted by updating the gate output coefficient. Correspondingly, the update gate is used to control the fusion ratio between the historical state and the current state, which is determined by the slope of the sharpness change. If the absolute value of the sharpness slope is large, the focusing trend is obvious, the larger the update gate output coefficient is, and the current state is updated first. If the sharpness slope is close to 0, it means that the sharpness is close to the peak value, the smaller the update gate output coefficient is, and the historical state is retained.

[0100] Furthermore, the final hidden state can be determined based on the candidate hidden state and the second output value.

[0101] Correspondingly, the final hidden state can be obtained through equation (7):

[0102] (7)

[0103] in, This is the final hidden state. Update the gate's output value at the current moment, i.e., the second output value.

[0104] Furthermore, the trend correction coefficient can be determined through the final hidden state.

[0105] The predicted focus distance is determined based on the trend correction coefficient and the slope of sharpness change. This process can be implemented as follows: obtain the cumulative focus offset and the initial focus distance, where the cumulative focus offset is the cumulative focus offset from the beginning of the focusing process to the previous iteration, and the initial focus distance is the focus distance of the first frame image captured during the focusing process. The cumulative focus offset can be updated by the trend correction coefficient and the slope of sharpness change. The specific process is shown in Equation (8):

[0106] (8)

[0107] in, This is the updated focus offset cumulative amount. This is the cumulative focus offset before the update. The slope of the sharpness change. This is the trend correction coefficient.

[0108] Finally, by adding the updated cumulative focus offset to the initial focus distance, the predicted focus distance can be obtained, and thus the first confidence level of the predicted focus distance can be calculated.

[0109] In some embodiments, the incremental data vector also includes a second confidence level, which can be used to determine the predicted focus distance and the first confidence level if the second confidence level is greater than the corresponding threshold.

[0110] In some embodiments, the training process of the convolutional neural network includes: acquiring first sample data, wherein the first sample data is any sample data in a first sample dataset, and each sample data in the first sample dataset includes: a sample image, a sample focus offset of the sample image, sample image features, and a sample image sharpness value; inputting the first sample data into the convolutional neural network to obtain the predicted sample focus offset of the first sample image, the predicted sample image features of the first sample image, and the predicted sample image sharpness value; calculating a first loss function value based on the sample focus offset and the predicted sample focus offset; calculating a second loss function value based on the sample image features and the predicted sample image features; calculating a third loss function value based on the sample image sharpness value and the predicted sample image sharpness value; adjusting the model parameters of the convolutional neural network based on the first loss function value, the second loss function value, and the third loss function value, and continuing training until the first training completion condition is met to obtain a trained convolutional neural network.

[0111] In some embodiments, the training process of the recurrent neural network includes: acquiring second sample data, which is any sample data in a second sample dataset, wherein each sample data in the second sample dataset includes: the focus offset, feature matching degree, sharpness change slope, and sample focus distance of the second sample image; inputting the second sample data into the recurrent neural network to obtain the predicted sample focus distance; calculating a fourth loss function value based on the sample focus distance and the predicted sample focus distance; adjusting the model parameters of the recurrent neural network based on the fourth loss function value, and continuing training until the second training completion condition is met to obtain a trained recurrent neural network.

[0112] In some embodiments, a total loss function value can be calculated based on the first loss function value, the second loss function value, and the third loss function value. The model parameters of the convolutional neural network can be adjusted based on the total loss function value, and training can continue until the first training completion condition is met, resulting in a trained convolutional neural network.

[0113] The first and second training completion conditions can be flexibly set based on the actual situation, such as the number of training sessions reaching a threshold, or the total loss function value being less than a preset loss function value.

[0114] In some embodiments, to provide a detailed explanation of the above image processing method, this application also provides a flowchart of the image processing method, such as... Figure 3 As shown, the method includes the following steps.

[0115] While in the focusing process and acquiring the current image captured by the imaging device, S301 is executed to acquire the current image and input it into the trained convolutional neural network to obtain the first prediction result and the previous frame image and input it into the trained convolutional neural network to obtain the second prediction result.

[0116] S302, Obtain the first predicted focus offset, the first sharpness value, and the first image feature of the first prediction result, as well as the second sharpness value and the second image feature of the second prediction result.

[0117] S303. Construct time-series correlation parameters based on the first sharpness value, the first image feature, the second sharpness value, and the second image feature.

[0118] S304. Construct the first predicted focus offset and time-series correlation parameters to obtain the incremental data vector.

[0119] S305. Input the incremental data vector into the trained recurrent neural network to obtain the third prediction result, which includes the predicted focus distance and the first confidence level.

[0120] S306. Determine whether the focusing completion conditions are met.

[0121] If the first focusing condition is met, execute S307 to control the imaging device to capture the target image based on the predicted focusing distance.

[0122] The first focus completion condition is that the first confidence level is greater than the preset confidence level threshold.

[0123] If the second focus completion condition is met, then execute S308 to determine the target confidence level from the multiple first confidence levels output by the trained recurrent neural network during the focus process.

[0124] The second condition for completing focus is that the number of shots corresponding to the current image reaches a preset shooting threshold.

[0125] S309. Obtain the target prediction focusing distance corresponding to the target credibility, and control the imaging device to capture the target image based on the target prediction focusing distance.

[0126] If neither the first focus completion condition nor the second focus completion condition is met, execute S310 to obtain the first predicted focus offset of the first prediction result and the first focus distance corresponding to the current image.

[0127] S311. Determine the deviation direction based on the predicted focusing distance and the first focusing distance.

[0128] S312. Determine the second focus distance based on the first predicted focus offset and the first focus distance.

[0129] S313. Acquire the next frame image captured by the imaging device based on the second focusing distance, and return to S301.

[0130] This application also provides an image processing apparatus for performing the image processing method provided in any of the above embodiments. For example... Figure 4 As shown, the device includes: an execution module 401, an acquisition module 402, a construction module 403, an input module 404, and a control module 405.

[0131] Execution module 401 is configured to perform the following iterative process while in the focusing process and acquiring the current image captured by the imaging device:

[0132] The acquisition module 402 is used to acquire the first prediction result obtained by inputting the current image into the trained convolutional neural network and the second prediction result obtained by inputting the previous frame image into the trained convolutional neural network;

[0133] Construction module 403 is used to construct an incremental data vector based on the first prediction result and the second prediction result;

[0134] The input module 404 is used to input the incremental data vector into the trained recurrent neural network to obtain a third prediction result, the third prediction result including: predicted focus distance and first confidence level;

[0135] The control module 405 is used to complete the iterative process if the first focusing completion condition is met, and to control the imaging device to capture the target image based on the predicted focusing distance. The first focusing completion condition is that the first confidence level is greater than a preset confidence level threshold.

[0136] This application proposes an image processing apparatus. In the process of focusing, a trained convolutional neural network outputs a first prediction result for the current image and a second prediction result for the previous frame image. An incremental data vector is constructed based on the first and second prediction results. This incremental data vector is then input into the trained recurrent neural network. This allows the recurrent neural network to improve focusing accuracy by incorporating temporal changes across multiple frames when determining the predicted focus distance. Furthermore, after each frame is captured, an incremental data vector is input into the recurrent neural network in real time to dynamically update the predicted focus distance. The system dynamically terminates shooting based on confidence level and the number of shots, eliminating the need to wait for complete multi-frame images and reducing redundant shooting.

[0137] In some embodiments, the above-described apparatus further includes: a determining module;

[0138] The determination module is used to complete the iterative process if the second focus completion condition is met, and to determine the target confidence from multiple first confidence values ​​output by the trained recurrent neural network in the focus process. The second focus completion condition is that the number of shots corresponding to the current image reaches a preset number of shots threshold.

[0139] The acquisition module 402 is also used to acquire the target prediction focusing distance corresponding to the target confidence level, and control the imaging device to capture a target image based on the target prediction focusing distance.

[0140] In some embodiments:

[0141] The acquisition module 402 is further configured to acquire the first predicted focus offset of the first prediction result and the first focus distance corresponding to the current image if the first focus completion condition and the second focus completion condition are not met.

[0142] The determining module is further configured to determine the deviation direction based on the predicted focusing distance and the first focusing distance;

[0143] The determining module is further configured to determine a second focusing distance based on the first predicted focusing offset and the first focusing distance;

[0144] The acquisition module 402 is further configured to acquire the next frame image captured by the imaging device based on the second focusing distance;

[0145] The input module 404 is further configured to input the next frame image and the deviation direction into the trained convolutional neural network and continue iterating until the first focus completion condition or the second focus completion condition is met.

[0146] In some embodiments, the first prediction result further includes a first sharpness value and a first image feature, and the second prediction result includes a second sharpness value and a second image feature. The construction module 403 is specifically used for:

[0147] Calculate the feature matching degree based on the first image features and the second image features;

[0148] Obtain the third focus distance of the previous frame image;

[0149] The focus distance difference is calculated based on the first focus distance and the third focus distance;

[0150] Calculate the sharpness difference based on the first sharpness value and the second sharpness value;

[0151] The slope of the sharpness change is determined based on the sharpness difference and the focus distance difference;

[0152] The feature matching degree and the sharpness change slope are combined to obtain the temporal correlation parameters;

[0153] An incremental data vector is obtained by constructing the first predicted focus offset and the time-series correlation parameters.

[0154] In some embodiments, the process of determining the first sharpness value includes:

[0155] Divide the current image into multiple regions;

[0156] Calculate the grayscale threshold for each region and the average grayscale threshold for all regions;

[0157] The region with a grayscale threshold greater than the average grayscale threshold is defined as the target region;

[0158] Calculate the grayscale variance for each target region;

[0159] If the grayscale variance of the first target region is less than the grayscale threshold of the first target region, then the grayscale gradient of the first target region is set to 0, and the first target region is any target region among the target regions.

[0160] If the grayscale variance of the first target region is greater than or equal to the grayscale threshold of the first target region, then the grayscale gradient of the first target region is calculated.

[0161] A first sharpness value is determined based on the grayscale gradient of each target region.

[0162] In some embodiments, the second prediction result further includes a second predicted focus offset. The trained recurrent neural network is equipped with a gated recurrent unit, which includes a reset gate and an update gate. The input module 404 is specifically used for:

[0163] The first output value of the reset gate is determined based on the feature matching degree;

[0164] The second output value of the update gate is determined based on the slope of the sharpness change.

[0165] The candidate hidden state is determined based on the first output value, the first predicted focus offset, and the second predicted focus offset;

[0166] The final hidden state is determined based on the candidate hidden states and the second output value;

[0167] The predicted focus distance is determined based on the final hidden state and the slope of the sharpness change.

[0168] Calculate the first confidence level of the predicted focus distance;

[0169] A third prediction result is obtained based on the predicted focus distance and the first confidence level.

[0170] In some embodiments, the training process of the convolutional neural network includes:

[0171] Obtain first sample data, which is any sample data in the first sample dataset. Each sample data in the first sample dataset includes: a sample image, a sample focus offset of the sample image, sample image features, and sample image sharpness value.

[0172] The first sample data is input into a convolutional neural network to obtain the predicted sample focus offset of the first sample image, the predicted sample image features of the first sample image, and the predicted sample image sharpness value.

[0173] The first loss function value is calculated based on the sample focus offset and the predicted sample focus offset;

[0174] Calculate the second loss function value based on the features of the sample image and the features of the predicted sample image;

[0175] The third loss function value is calculated based on the sharpness value of the sample image and the sharpness value of the predicted sample image;

[0176] The model parameters of the convolutional neural network are adjusted based on the first loss function value, the second loss function value, and the third loss function value, and training continues until the first training completion condition is met, resulting in a trained convolutional neural network.

[0177] This application also provides an electronic device for performing the above-described image processing method. Please refer to... Figure 5 It illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 5 As shown, the electronic device 5 includes: a processor 500, a memory 501, a bus 502 and a communication interface 503. The processor 500, the communication interface 503 and the memory 501 are connected through the bus 502. The memory 501 stores a computer program that can run on the processor 500. When the processor 500 runs the computer program, it executes the image processing method provided in any of the foregoing embodiments of this application.

[0178] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0179] Bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. Memory 501 is used to store programs. After receiving execution instructions, processor 500 executes the programs. The image processing method disclosed in any of the aforementioned embodiments of this application can be applied to processor 500, or implemented by processor 500.

[0180] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0181] The electronic device provided in this application embodiment and the image processing method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0182] This application also provides a computer-readable storage medium corresponding to the image processing method provided in the foregoing embodiments. Please refer to... Figure 6 The computer-readable storage medium shown is an optical disc 60, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the image processing method provided in any of the foregoing embodiments.

[0183] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0184] The computer-readable storage medium provided in the above embodiments of this application and the image processing method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0185] It should be noted that:

[0186] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0187] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0188] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0189] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, include: While focusing and acquiring the current image captured by the imaging device, the following iterative process is performed: The current image is input into the trained convolutional neural network to obtain a first prediction result, and the previous frame image is input into the trained convolutional neural network to obtain a second prediction result; An incremental data vector is constructed based on the first prediction result and the second prediction result; The incremental data vector is input into the trained recurrent neural network to obtain a third prediction result, which includes a predicted focus distance and a first confidence level, wherein the first confidence level is used to reflect the reliability of the predicted focus distance. If the first focus completion condition is met, the iteration process is completed, and the imaging device is controlled to capture the target image based on the predicted focus distance. The first focus completion condition is that the first confidence level is greater than the preset confidence level threshold. The method further includes: If the second focus completion condition is met, the iteration process is completed. The target confidence level is determined from the multiple first confidence levels output by the trained recurrent neural network in the focus process. The second focus completion condition is that the number of shots corresponding to the current image reaches a preset number of shots threshold. Obtain the target prediction focus distance corresponding to the target confidence level, and control the imaging device to capture a target image based on the target prediction focus distance; The method further includes: If neither the first focus completion condition nor the second focus completion condition is met, then obtain the first predicted focus offset of the first prediction result and the first focus distance corresponding to the current image. The deviation direction is determined based on the predicted focusing distance and the first focusing distance; The second focus distance is determined based on the first predicted focus offset and the first focus distance; Acquire the next frame image captured by the imaging device based on the second focusing distance; The next frame image and the deviation direction are input into the trained convolutional neural network, and the iteration continues until the first focus completion condition or the second focus completion condition is met. The first prediction result further includes a first sharpness value and a first image feature, and the second prediction result includes a second sharpness value and a second image feature. The step of constructing an incremental data vector based on the first prediction result and the second prediction result includes: Calculate the feature matching degree based on the first image features and the second image features; Obtain the third focus distance of the previous frame image; The focus distance difference is calculated based on the first focus distance and the third focus distance; Calculate the sharpness difference based on the first sharpness value and the second sharpness value; The slope of the sharpness change is determined based on the sharpness difference and the focus distance difference; The feature matching degree is used to reflect the background changes between the current image and the previous frame image, and the sharpness change slope is used to reflect the sharpness changes between the current image and the previous frame image; The feature matching degree and the sharpness change slope are combined to obtain the temporal correlation parameters; An incremental data vector is obtained by constructing the first predicted focus offset and the temporal correlation parameters; The second prediction result also includes a second predicted focus offset. The trained recurrent neural network is equipped with a gated recurrent unit, which includes a reset gate and an update gate. The step of inputting the incremental data vector into the trained recurrent neural network to obtain the third prediction result includes: The first output value of the reset gate is determined based on the feature matching degree; The second output value of the update gate is determined based on the slope of the sharpness change. The candidate hidden state is determined based on the first output value, the first predicted focus offset, and the second predicted focus offset; The final hidden state is determined based on the candidate hidden states and the second output value; The predicted focus distance is determined based on the final hidden state and the slope of the sharpness change. Calculate the first confidence level of the predicted focus distance; A third prediction result is obtained based on the predicted focus distance and the first confidence level.

2. The method according to claim 1, characterized in that, The process of determining the first sharpness value includes: Divide the current image into multiple regions; Calculate the grayscale threshold for each region and the average grayscale threshold for all regions; The region with a grayscale threshold greater than the average grayscale threshold is defined as the target region; Calculate the grayscale variance for each target region; If the grayscale variance of the first target region is less than the grayscale threshold of the first target region, then the grayscale gradient of the first target region is set to 0, and the first target region is any target region among the target regions. If the grayscale variance of the first target region is greater than or equal to the grayscale threshold of the first target region, then the grayscale gradient of the first target region is calculated. A first sharpness value is determined based on the grayscale gradient of each target region.

3. The method of claim 1, wherein, The training process of the convolutional neural network includes: Obtain first sample data, which is any sample data in the first sample dataset. Each sample data in the first sample dataset includes: a sample image, a sample focus offset of the sample image, sample image features, and sample image sharpness value. The first sample data is input into a convolutional neural network to obtain the predicted sample focus offset of the first sample image, the predicted sample image features of the first sample image, and the predicted sample image sharpness value. The first loss function value is calculated based on the sample focus offset and the predicted sample focus offset; Calculate the second loss function value based on the features of the sample image and the features of the predicted sample image; The third loss function value is calculated based on the sharpness value of the sample image and the sharpness value of the predicted sample image; The model parameters of the convolutional neural network are adjusted based on the first loss function value, the second loss function value, and the third loss function value, and training continues until the first training completion condition is met, resulting in a trained convolutional neural network.

4. An image processing apparatus characterized by comprising: include: The execution module is used to perform the following iterative process while the focusing process is in progress and the current image captured by the imaging device is being acquired: The acquisition module is used to acquire the first prediction result obtained by inputting the current image into the trained convolutional neural network and the second prediction result obtained by inputting the previous frame image into the trained convolutional neural network; A construction module is used to construct an incremental data vector based on the first prediction result and the second prediction result; An input module is used to input the incremental data vector into a trained recurrent neural network to obtain a third prediction result. The third prediction result includes a predicted focus distance and a first confidence level, wherein the first confidence level is used to reflect the reliability of the predicted focus distance. The control module is used to complete the iterative process if the first focus completion condition is met, and to control the imaging device to capture the target image based on the predicted focus distance. The first focus completion condition is that the first confidence level is greater than a preset confidence level threshold. The aforementioned device also includes: a determining module; The determining module is used to complete the iterative process if the second focusing completion condition is met, and to determine the target confidence from multiple first confidence values ​​output by the trained recurrent neural network in the focusing process. The second focusing completion condition is that the number of shots corresponding to the current image reaches a preset number of shots threshold. The acquisition module is further configured to acquire the target prediction focusing distance corresponding to the target confidence level, and control the imaging device to capture a target image based on the target prediction focusing distance; The acquisition module is further configured to acquire the first predicted focus offset of the first prediction result and the first focus distance corresponding to the current image if the first focus completion condition and the second focus completion condition are not met. The determining module is further configured to determine the deviation direction based on the predicted focusing distance and the first focusing distance; The determining module is further configured to determine a second focusing distance based on the first predicted focusing offset and the first focusing distance; The acquisition module is further configured to acquire the next frame image captured by the imaging device based on the second focusing distance; The input module is further configured to input the next frame image and the deviation direction into the trained convolutional neural network and continue iterating until the first focus completion condition or the second focus completion condition is met. The first prediction result further includes a first sharpness value and a first image feature, and the second prediction result includes a second sharpness value and a second image feature. The construction module is specifically used for: Calculate the feature matching degree based on the first image features and the second image features; Obtain the third focus distance of the previous frame image; The focus distance difference is calculated based on the first focus distance and the third focus distance; Calculate the sharpness difference based on the first sharpness value and the second sharpness value; The slope of the sharpness change is determined based on the sharpness difference and the focus distance difference; The feature matching degree is used to reflect the background changes between the current image and the previous frame image, and the sharpness change slope is used to reflect the sharpness changes between the current image and the previous frame image; The feature matching degree and the sharpness change slope are combined to obtain the temporal correlation parameters; An incremental data vector is obtained by constructing the first predicted focus offset and the temporal correlation parameters; The second prediction result also includes a second predicted focus offset. The trained recurrent neural network is equipped with a gated recurrent unit, which includes a reset gate and an update gate. The input module is specifically used for: The first output value of the reset gate is determined based on the feature matching degree; The second output value of the update gate is determined based on the slope of the sharpness change. The candidate hidden state is determined based on the first output value, the first predicted focus offset, and the second predicted focus offset; The final hidden state is determined based on the candidate hidden states and the second output value; The predicted focus distance is determined based on the final hidden state and the slope of the sharpness change. Calculate the first confidence level of the predicted focus distance; A third prediction result is obtained based on the predicted focus distance and the first confidence level.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the method as described in any one of claims 1-3.

Images