Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring initial image data and motion data corresponding to the initial image data; determining an anti-shake compensation amount corresponding to the initial image data according to the motion data; and inputting the initial image data and the anti-shake compensation amount into an optical anti-shake model to obtain a target image corresponding to the initial image data. By adopting the method, the anti-shake compensation effect can be enhanced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of electronic devices such as smart phones, shooting technology has become increasingly popular. People can use cameras to take photos or videos anytime and anywhere. However, in the process of using the camera, the electronic device often shakes due to external influences, resulting in blurred images. Therefore, it is necessary to perform anti-shake processing on the electronic device.

[0003] Optical Image Stabilization (OIS) is a common anti-shake processing method. It detects lens shake through a sensor and then performs hardware compensation for the lens shake, but the effect of anti-shake compensation is not good. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, an apparatus, an electronic device, and a computer-readable storage medium, which can enhance the effect of anti-shake compensation.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Acquire initial image data and motion data corresponding to the initial image data;

[0007] determining an anti-shake compensation amount corresponding to the initial image data according to the motion data;

[0008] The initial image data and the anti-shake compensation amount are input into an optical anti-shake model to obtain a target image corresponding to the initial image data.

[0009] In a second aspect, the present application further provides an image processing device, comprising:

[0010] A data acquisition module, used to acquire initial image data and motion data corresponding to the initial image data;

[0011] an anti-shake compensation amount determination module, configured to determine the anti-shake compensation amount corresponding to the initial image data according to the motion data;

[0012] The anti-shake compensation module is used to input the initial image data and the anti-shake compensation amount into an optical anti-shake model to obtain a target image corresponding to the initial image data.

[0013] In a third aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image processing method provided in the first aspect when executing the computer program.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image processing method provided in the first aspect are implemented.

[0015] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image processing method provided in the first aspect.

[0016] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product obtain initial image data and motion data corresponding to the initial image data, determine the anti-shake compensation amount corresponding to the initial image data according to the motion data, input the initial image data and the anti-shake compensation amount into the optical anti-shake model, and obtain the target image corresponding to the initial image data. It is possible to suppress the motion blur in the initial image data through the optical anti-shake model to obtain a target image with high clarity and stable object, and there is no restriction on the anti-shake angle, and anti-shake at a larger shake angle can be achieved, thereby enhancing the anti-shake compensation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0019] Figure 2 is a schematic flow chart of an image processing method in one embodiment;

[0020] Figure 3 is a flow chart of step 206 in one embodiment;

[0021] Figure 4 is a schematic diagram of aligning motion information with standard motion information in one embodiment;

[0022] Figure 5 is a schematic diagram of a process of obtaining a target image through an optical image stabilization model in one embodiment;

[0023] Figure 6 is a schematic structural diagram of an optical image stabilization model in an embodiment;

[0024] Figure 7 is a schematic flow chart of an image processing method in another embodiment;

[0025] Figure 8 is a schematic flow chart of an image processing method in another embodiment;

[0026] Fig. 9 is a schematic flow chart of an image processing method in another embodiment;

[0027] Fig.10 Schematic diagram of a training process of an optical image stabilization model in one embodiment;

[0028] Fig.11 A schematic diagram of a process for obtaining positive sample image data and negative sample image data under the same motion trajectory in one embodiment;

[0029] Fig.12 A schematic diagram of a model selection interface in an embodiment;

[0030] Fig.13A is a schematic diagram of various components and their associated functions of a model management system 1300 for assigning appropriate AI models based on user specifications in one embodiment;

[0031] Fig. 13B is a schematic diagram of additional components and associated functions of a model management system 1300 for training an AI model pool in one embodiment;

[0032] Fig.14 is a flow chart of an implementation scheme of a testing process for a neural network module in one embodiment;

[0033] Fig.15 A flowchart of a neural network module implementing a facial detection process in one embodiment;

[0034] Fig.16 is a structural block diagram of an image processing device in one embodiment;

[0035] Fig.17 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] Traditional OIS compensation calculates the anti-shake compensation amount of the lens through sensor data, and then uses PID (proportion integration differentiation) control logic to drive the IC (Integrated Circuit) to drive the motor to push the lens, so that the lens moves the distance corresponding to the anti-shake compensation amount, and the sensor outputs the image after OIS compensation. However, the lens movement distance is limited, and the corresponding shake compensation angle is also limited. If the shake angle is large, the corresponding anti-shake compensation effect is poor.

[0038] Based on the technical problem that the traditional OIS compensation has a poor anti-shake compensation effect, the embodiment of the present application provides an image processing method, which achieves an enhanced anti-shake compensation effect by using an optical anti-shake model to perform shake compensation.

[0039] The image processing method provided in the embodiment of the present application can be applied to Figure 1 The electronic device shown in the figure. The electronic device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. It can be understood that the electronic device may include a terminal or a server.

[0040] In some exemplary embodiments, Figure 2 As shown, an image processing method is provided, comprising the following steps 202 to 206. Among them:

[0041] Step 202: Acquire initial image data and motion data corresponding to the initial image data.

[0042] Among them, the initial image data refers to the image data to be deblurred. The initial image data can be image data collected when the OIS component is disabled or enabled, or it can be image data collected by a camera module without an OIS component installed. In other words, the initial image data can be blurred image data or clear image data. The initial image data can be image data in the raw domain, YUV domain, or RGB (red, green, and blue) domain. Generally, the greater the image depth (color depth) corresponding to the initial image data, the clearer the target image obtained in the end. For example, the acquired initial image data can be raw domain data with a color depth of 10bit or 12bit. In actual application scenarios, the initial image data can be video frame data in a video stream, and each frame of video frame data in the video stream is used as the initial image data.

[0043] Each frame of the initial image corresponds to corresponding motion data, wherein the motion data refers to data collected by the inertial sensor, and the motion data is used to characterize the motion trajectory of the lens. The motion data may include angular velocity data and / or acceleration data, wherein the angular velocity data is collected by an angular velocity sensor, such as a gyroscope, and the acceleration data is collected by an acceleration sensor, such as an accelerometer.

[0044] Exemplarily, the image sensor in the camera module can be used to collect initial image data in the raw domain, and then obtain motion data corresponding to the initial image data. Since the image acquisition frequency is often different from the acquisition frequency of the inertial sensor, it is necessary to align the frame time of the image data with the acquisition time of the motion data.

[0045] Step 204: determine the anti-shake compensation amount corresponding to the initial image data according to the motion data.

[0046] The anti-shake compensation refers to the compensation amount corresponding to overcoming blur caused by shaking, and the anti-shake compensation amount can be characterized by displacement compensation amount and / or angle compensation amount. The anti-shake compensation can be to move the lens toward the optical center, and the anti-shake compensation amount represents the movement distance of the lens toward the optical center.

[0047] Optionally, the motion data includes angular velocity data, and the shake angle can be determined by the angular velocity data, for example, by filtering, integrating, etc. the angular velocity data to obtain the shake angle, thereby determining the angle compensation amount according to the shake angle. Alternatively, the motion data includes acceleration data, and the acceleration data can be filtered, integrated, etc. to obtain the shake displacement, thereby determining the displacement compensation amount according to the shake displacement. Alternatively, the anti-shake compensation amount corresponding to the initial image data can also be determined jointly by the angular velocity data and the acceleration data.

[0048] Step 206 , inputting the initial image data and the anti-shake compensation amount into the optical anti-shake model to obtain a target image corresponding to the initial image data.

[0049] Among them, the optical image stabilization model refers to a model that performs jitter compensation on the initial image data based on the anti-shake compensation amount. The optical image stabilization model can clarify the initial image data and stabilize the object in the initial image data. Optionally, the optical image stabilization model can be trained based on an image training set, wherein the image training set includes a plurality of sample image pairs, each sample image pair includes a frame of clear image and a frame of blurred image, and the frame of clear image and the frame of blurred image are collected under the same motion trajectory. The object in the clear image is stable.

[0050] The target image is an image obtained by performing anti-shake compensation on the initial image data, that is, the target image is an anti-shake image, and the clarity of the target image is higher than the clarity of the initial image data.

[0051] In some optional application scenarios, the shutter button is pressed to shoot, and the image sensor collects multiple frames of images to be processed. The initial image data and anti-shake compensation amount corresponding to each frame of the image to be processed can be input into the optical anti-shake model in sequence to obtain the target image corresponding to each frame of the image to be processed, and then the target images corresponding to each frame of the image to be processed are fused to obtain the final captured image.

[0052] The above-mentioned image processing method obtains initial image data and motion data corresponding to the initial image data, determines the anti-shake compensation amount corresponding to the initial image data according to the motion data, inputs the initial image data and the anti-shake compensation amount into the optical anti-shake model, and obtains the target image corresponding to the initial image data. It can suppress the motion blur in the initial image data through the optical anti-shake model, and obtain a target image with high clarity and a relatively stable object. At the same time, there is no restriction on the anti-shake angle, and anti-shake at a larger shake angle can be achieved, thereby enhancing the anti-shake compensation effect.

[0053] In some embodiments, Figure 3 As shown, the initial image data and the anti-shake compensation amount are input into the optical anti-shake model to obtain the step 206 of the target image corresponding to the initial image data, including steps 302 to 306.

[0054] Step 302: Perform motion prediction on the anti-shake compensation amount by using an optical anti-shake model to obtain motion information.

[0055] Motion prediction refers to the prediction of the motion information of a lens. The motion information can be represented by a motion vector. The motion information includes the motion distance and the motion direction.

[0056] Optionally, after obtaining the motion information, the motion information can be aligned with the standard motion information under the standard state to obtain the aligned motion information. The standard motion information refers to the motion information under the standard state. The standard state can be the state where the center of the lens is at the optical center. It is easy to understand that the standard motion information corresponding to the center of the lens at the optical center is (0,0), that is, the motion distance and the motion direction are both 0. The standard state can also be set according to the actual application scenario. For example, Figure 4 As shown, assuming that in the initial state, that is, when preparing to shoot, the position of the lens is at the optical center, and the corresponding motion information is (0,0), the initial state is taken as the standard state, and the motion information in the initial state is taken as the standard motion information. During the shooting process, the motion information corresponding to each subsequent frame of image data is aligned with the standard motion information. Aligning the motion information corresponding to the initial image data with the standard motion information is essentially mapping the motion information to the standard motion information, thereby obtaining the aligned running information.

[0057] Step 304: Deblur the initial image data to obtain deblurred image data.

[0058] Among them, the deblurring process is equivalent to improving the clarity of the image, and the clarity of the deblurred image data is higher than the clarity of the initial image data.

[0059] Optionally, the initial image data may be subjected to fuzzy prediction through a fuzzy prediction network to obtain a deblurring kernel, and then the initial image data may be deblurred based on the deblurring kernel to obtain deblurred image data.

[0060] Optionally, the initial image data may be deblurred based on the motion information to obtain deblurred image data. Blur prediction may be performed based on the initial image data and the motion information to obtain a deblur kernel, and then the initial image data may be deblurred based on the deblur kernel to obtain deblurred image data. For example, a convolution calculation may be performed on the motion information to obtain motion convolution data, the motion convolution data may be concatenated with the initial image data to obtain blur prediction data, blur prediction may be performed based on the blur prediction data to obtain a deblur kernel corresponding to the initial image data, and the deblur kernel may be multiplied with the initial image data to obtain the deblurred image data.

[0061] Step 306: Reconstruct the image based on the deblurred image data and the motion information to obtain a target image.

[0062] Optionally, the deblurred image data and the aligned motion information can be reconstructed through a reconstruction network to obtain a residual image, and the target image can be obtained based on the residual image and the initial image data.

[0063] For example, the deblurred image data and the aligned motion information may be concatenated to obtain concatenated image data, and the concatenated image data may be reconstructed through a reconstruction network to obtain an image residual, and the image residual may be summed with the initial image data to obtain a target image. The concatenation may include splicing, such as splicing of image data channels.

[0064] In an optional example, a schematic diagram of obtaining a target image through an optical image stabilization model is as follows: Figure 5 As shown, the initial image data and the corresponding anti-shake compensation amount are input into the optical anti-shake model, and the motion of the anti-shake compensation amount is predicted by the optical anti-shake model to obtain motion information, and the initial image data is deblurred to obtain deblurred image data. It should be noted that the initial image data can be directly deblurred to obtain deblurred image data, or the initial image data can be deblurred according to the motion information to obtain deblurred image data. Then, image reconstruction is performed according to the motion information and the deblurred image data to obtain the target image.

[0065] In this embodiment, the clarity of the target image can be improved by performing motion prediction on the anti-shake compensation amount to obtain motion information, deblurring the initial image data to obtain deblurred image data, and then obtaining the target image based on the motion information and the deblurred image data.

[0066] In some exemplary embodiments, the optical image stabilization model includes a motion prediction network, a blur prediction network, an alignment network, a deblurring network, and a reconstruction network; the initial image data and the image stabilization compensation amount are input into the optical image stabilization model to obtain a target image corresponding to the initial image data, including:

[0067] The motion prediction network is used to perform motion prediction on the anti-shake compensation amount to obtain motion information; the alignment network is used to align the motion information with the standard motion information in the standard state to obtain the aligned motion information; the blur prediction network is used to perform blur prediction on the initial image data to obtain the deblur kernel; the deblur network is used to perform deblurring on the initial image data based on the deblur kernel to obtain the deblurred image data; the reconstruction network is used to reconstruct the deblurred image data and the aligned motion information to obtain the target image.

[0068] Among them, the meanings of the corresponding terms and steps in this embodiment can be found in the explanations and illustrations of the corresponding terms and steps in the above embodiments, and will not be repeated here.

[0069] In one example, the structure diagram of the optical image stabilization model is as follows: Figure 6As shown, the optical image stabilization model includes a motion prediction network, an alignment network, a blur prediction network, a deblurring network and a reconstruction network, wherein the motion prediction network is used to predict motion information, the alignment network is used to align the motion information with the standard motion information, the blur prediction network is used to predict the deblurring kernel of the initial image data, the deblurring network is used to perform deblurring processing based on the deblurring kernel, and the reconstruction network is used to reconstruct the image. Inputting the initial image data and the anti-shake compensation amount into the optical anti-shake model, the processing flow of obtaining the target image corresponding to the initial image data includes: performing motion prediction on the anti-shake compensation amount through a motion prediction network to obtain motion information; aligning the motion information with the standard motion information through an alignment network to obtain aligned motion information; performing convolution operation on the motion information to obtain motion convolution data, and concatenating the motion convolution data with the initial image data to obtain blur prediction data; performing blur prediction on the blur prediction data through a blur prediction network to obtain a deblur kernel; performing deblurring processing on the initial image data based on the deblur kernel through a deblur network to obtain deblurred image data; concatenating the aligned motion information and the deblurred image data to obtain concatenated image data, reconstructing the concatenated image data through a reconstruction network to obtain an image residual, and summing the image residual with the initial image data to obtain a target image.

[0070] In this embodiment, the motion prediction network, blur prediction network, alignment network, deblurring network and reconstruction network included in the optical image stabilization model perform corresponding processing respectively to obtain the target image, that is, each network processes the corresponding task respectively without affecting each other, which can improve the processing accuracy and efficiency of the optical image stabilization model.

[0071] In some embodiments, the motion data includes angular velocity data and acceleration data; the step 204 of determining the anti-shake compensation amount corresponding to the initial image data according to the motion data includes:

[0072] The shaking angle is determined according to the angular velocity data; the shaking displacement is determined according to the acceleration data; and the anti-shake compensation amount corresponding to the initial image data is determined based on the shaking angle and the shaking displacement.

[0073] The shake angle refers to the rotation angle of the lens during the shake process, and the shake displacement refers to the movement displacement of the lens during the shake process.

[0074] For example, the angular velocity data may be filtered and integrated to obtain the shaking angle, and correspondingly, the acceleration data may be filtered and integrated to obtain the shaking displacement. The shaking displacement generally includes displacements on different coordinate axes, such as horizontal and vertical displacements.

[0075] Optionally, the jitter offset on the image plane caused by the jitter angle can be calculated according to the jitter angle, and the final anti-shake compensation amount corresponding to the initial image data can be determined according to the jitter offset and the jitter displacement. For example, the jitter offset can be obtained according to the product of the focal length and the jitter angle, and the jitter displacement can be obtained according to the jitter displacement in the horizontal direction and the jitter displacement in the vertical direction based on the Pythagorean theorem. The final anti-shake compensation amount can be calculated according to the jitter offset and the jitter displacement according to the Pythagorean theorem, that is, the sum of the square of the jitter offset and the square of the jitter displacement is calculated to obtain the square sum, and then the square root of the square sum is calculated to obtain the anti-shake compensation amount.

[0076] In this embodiment, by jointly determining the anti-shake compensation amount corresponding to the initial image data according to the shake angle and the shake displacement, a more accurate anti-shake compensation amount can be obtained, thereby improving the clarity of the target image.

[0077] In some embodiments, obtaining motion data corresponding to the initial image data in step 202 includes:

[0078] If there is a moment in the collection of motion data that is the same as the frame moment of the initial image data, the motion data corresponding to the frame moment is used as the motion data corresponding to the initial image; if there is no moment in the collection of motion data that is the same as the frame moment of the initial image data, then among the collection moments of motion data, at least two collection moments closest to the frame moment of the initial image data are selected; based on the at least two collection moments and the motion data corresponding to the at least two collection moments, the motion data corresponding to the frame moment of the initial image data is determined.

[0079] It is easy to understand that the image acquisition frequency of the image sensor and the acquisition frequency of the inertial sensor in the same electronic device are usually different. When the image sensor acquires multiple frames of images, the frame time corresponding to each frame of the image is not necessarily the same as the acquisition time of the motion data. When obtaining the motion data corresponding to the initial image data, it is necessary to align the frame time of the initial image data and the acquisition time of the motion data to ensure that the acquired motion data corresponds to the initial image data.

[0080] In some application scenarios, the electronic device can record the frame time of collecting each frame of image data, as well as the sensor data of the inertial sensor within a preset time length and the collection time corresponding to the sensor data. The frame time refers to the collection time of the image frame. For the initial image data, check whether there is a time in the collection time of the motion data that is the same as the frame time of the initial image data; if there is a time in the collection time of the motion data that is the same as the frame time of the initial image data, the motion data corresponding to the frame time is used as the motion data corresponding to the initial image; the motion data corresponding to the frame time refers to the motion data collected before the frame time, including the motion data from the start time of collecting the motion data to the frame time.

[0081] If there is no moment in the acquisition moments of the motion data that is the same as the frame moment of the initial image data, then at least two acquisition moments closest to the frame moment are selected from the acquisition moments of the motion data. It should be noted that the at least two acquisition moments closest to the frame moment may be acquisition moments before or after the frame moment. Optionally, a straight line fitting or a curve fitting is performed based on the at least two acquisition moments and the motion data corresponding to the at least two acquisition moments to obtain a fitting straight line or a fitting curve, and then the motion data corresponding to the frame moment is obtained based on the fitting straight line or the fitting curve and the frame moment.

[0082] Exemplarily, if the selected acquisition moments include two, a straight line fitting is performed based on the two selected acquisition moments and the motion data corresponding to the two acquisition moments to obtain a fitting straight line, and the motion data corresponding to the frame moment is obtained based on the frame moment and the fitting straight line, thereby obtaining the motion data corresponding to the initial image data. Alternatively, if the selected acquisition moments include three, a curve fitting is performed based on the three selected acquisition moments and the motion data corresponding to the three acquisition moments to obtain a fitting curve, and the motion data corresponding to the frame moment can be obtained based on the fitting curve and the frame moment, thereby obtaining the motion data corresponding to the initial image data. The number of acquisition moments selected that are closest to the frame moment of the initial image data can be selected according to the actual application scenario, for example, it can be 2, 3, 4 or 5, etc.

[0083] In this embodiment, by searching whether there is a moment in the acquisition time of the motion data that is the same as the frame time of the initial image data, if not, then the motion data corresponding to the frame time is determined through at least two acquisition times closest to the frame time and the motion data corresponding to the at least two acquisition times, so that the initial image data and the motion data can be accurately aligned based on time, thereby improving the accuracy of the motion data corresponding to the initial image data, and then improving the clarity of the target image.

[0084] In some embodiments, the initial image data includes target video frame data; obtaining the initial image data and motion data corresponding to the initial image data includes: obtaining the target video frame data in the initial video stream and motion data corresponding to the target video frame data;

[0085] The above method further includes:

[0086] The target video stream corresponding to the initial video stream is obtained according to the target images corresponding to all the target video frame data; or, the target video stream corresponding to the initial video stream is obtained according to the video frame data other than the target video frame data in the initial video stream and the target images corresponding to all the target video frame data.

[0087] The target video frame data may be any frame of video frame data in the initial video stream. The initial video stream refers to the video stream to be deblurred, and the initial video stream may be a video stream acquired by a camera module without an OIS component installed, or a video stream acquired by a camera module with an OIS component installed, wherein the OIS component may be enabled or disabled.

[0088] In an actual application scenario, the target video frame data and the motion data corresponding to the target video frame data in the initial video stream are obtained, the anti-shake compensation amount corresponding to the target video frame data is determined according to the motion data corresponding to the target video frame data, and the target video frame data and the anti-shake compensation amount are input into the optical anti-shake model to obtain the target image corresponding to the target video frame data. Then, the target video stream corresponding to the initial video stream can be obtained according to the target images corresponding to all the target video frame data, sorted according to the frame acquisition timestamp, wherein the clarity of the target video stream is higher than the clarity of the initial video stream, and the number of frames of all the target video frame data can be less than or equal to the number of video frames in the initial video stream. For example, each frame of video frame data in the initial video stream can be used as the target video frame data in sequence, or the video frame data of each interval of one or two frames in the initial video stream can be used as the target video frame data. It should be noted that how to select the target video frame data from the initial video stream can be selected according to the actual application scenario, and is not specifically limited here.

[0089] Alternatively, the video frame data except the target video frame data in the initial video stream and the target images corresponding to all the target video frame data can be arranged in frame timestamp order to obtain the target video stream, that is, the target video stream includes the target images corresponding to all the target video frame data and the video frame images except the target video frame data in the initial video stream, and the number of frames of the target video frame data is less than the number of video frames in the initial video stream.

[0090] In this embodiment, the target video frame data in the initial video stream is obtained and anti-shake processing is performed through an optical anti-shake model to obtain a corresponding target image, and then a clear video stream is obtained based on the target image. This can process a video stream with shaking and blur into a clear video stream, thereby improving the clarity of the video stream and the stability of objects in the video.

[0091] In an exemplary embodiment, the flowchart of the image processing method is as follows: Figure 7 As shown, the electronic device obtains the initial image data and the motion data corresponding to the initial image data, calculates the anti-shake compensation amount corresponding to the motion data based on the OIS algorithm module, and then inputs the initial image data and the corresponding anti-shake compensation amount into the optical anti-shake model to obtain the target image corresponding to the initial image data, and the target image is an image after virtual optical anti-shake compensation. The initial image data can be blurred image data or clear image data. Blurred image data can be image data with a blur greater than a blur threshold, and correspondingly, clear image data can be image data with a blur not greater than a blur threshold.

[0092] This example can realize optical image stabilization compensation through the optical image stabilization model, obtain a stable and clear image of the object, without the limitation of the anti-shake angle, and can also realize anti-shake compensation in the case of large shaking, which can increase the application scenario range of optical image stabilization compensation and enhance the anti-shake compensation effect. In some embodiments, obtaining initial image data includes:

[0093] Initial image data obtained by performing optical image stabilization compensation through an optical image stabilization component is acquired.

[0094] The optical image stabilization compensation refers to the image stabilization compensation performed by an optical image stabilization component. In actual application scenarios, the electronic device may include an optical image stabilization component, and in the process of acquiring the initial image data, the optical image stabilization component is controlled to be enabled, so that the initial image data obtained is an image with image stabilization enabled.

[0095] In one example, the flowchart of the image processing method is as follows: Figure 8 As shown, optical image stabilization compensation is performed through an optical image stabilization component to obtain initial image data, and then based on the initial image data and corresponding motion data, a corresponding image stabilization compensation amount is calculated based on the motion data, and the initial image data and the image stabilization compensation amount are input into an optical image stabilization model to obtain a target image corresponding to the initial image data, and the target image can be understood as an image that has undergone virtual optical image stabilization compensation. Virtual optical image stabilization compensation can refer to optical image stabilization compensation implemented by software.

[0096] In this embodiment, by acquiring the initial image data of the optical image stabilization compensated by the physical component of the optical image stabilization, virtual optical image stabilization compensation is performed again through the optical image stabilization model based on the initial image data and the corresponding image stabilization compensation amount. It is possible to achieve image stabilization compensation for the angles that the optical image stabilization component cannot compensate for through the optical image stabilization model, and to achieve compensation for all-round shaking angles, thereby further improving the clarity of the target image and enhancing the image stabilization compensation effect.

[0097] In one exemplary embodiment, the initial image data includes blurred image data.

[0098] Wherein, when the initial image data is blurred image data, the camera module of the electronic device may not include an optical image stabilization component, or the camera module of the electronic device may include an optical image stabilization component, but the optical image stabilization component is disabled, that is, the optical image stabilization component is not in a working state, and the initial image data has not been processed by anti-shake compensation. Blurred image data may be image data whose blurriness is greater than a blurriness threshold.

[0099] In an optional application scenario, such as Fig. 9 As shown, the camera module that does not include an optical image stabilization component collects initial image data through an image sensor. The initial image data collected in this way may have motion blur and has not been processed for motion blur suppression. The image processor obtains the initial image data and the motion data corresponding to the initial image data, determines the anti-shake compensation amount corresponding to the initial image data according to the motion data, inputs the initial image data and the anti-shake compensation amount into the optical image stabilization model, obtains the target image corresponding to the initial image data, and the obtained target image is the image processed for anti-shake compensation. Among them, the optical image stabilization model can be set in the image processor.

[0100] In the present embodiment, when the initial image data is blurred image data, a clear image is obtained after being processed by the optical image stabilization model. There is no restriction on the anti-shake angle, and shake compensation can be achieved for any shake angle, thereby increasing the compensation range of the shake angle and enhancing the anti-shake compensation effect. At the same time, there is no need to install physical components for optical image stabilization, which can reduce hardware costs and hardware debugging costs, while freeing up space for electronic equipment and reducing the overall size of the electronic equipment.

[0101] In an exemplary embodiment, the method further includes: in response to a selection trigger of a target zoom ratio, determining a target focal length according to the target zoom ratio, and if the focal length corresponding to the target focal length is within a preset focal length range, inputting the initial image data and the anti-shake compensation amount into an optical anti-shake model to obtain a target image corresponding to the initial image data.

[0102] Among them, focal length refers to the segmentation of the focal length of the lens. Focal lengths usually include ultra-wide angle, wide angle, standard, medium focal length, medium-long focal length, long focal length, ultra-long focal length or other focal lengths, and the target focal length can be any focal length.

[0103] In actual application scenarios, users can adjust the zoom ratio by point-cut or slide-cut operations. The electronic device obtains the zoom ratio selected by the user, determines the target focal length according to the zoom ratio, and identifies the target focal length. If the target focal length is within the preset focal length range, optical image stabilization compensation is performed through the optical image stabilization model, that is, the initial image data and the image stabilization compensation amount are input into the optical image stabilization model to obtain the target image corresponding to the initial image data. The preset focal length range can be set according to the actual application scenario. For example, the preset focal length range is a zoom ratio less than or equal to 0.6 times the standard focal length. When the target focal length is within the preset focal length range, optical image stabilization compensation is not performed through the optical image stabilization component, but through the optical image stabilization model.

[0104] Optionally, if the focal length corresponding to the target focal segment is not within a preset focal length range, optical image stabilization compensation may be performed through an optical image stabilization component and / or optical image stabilization compensation may be performed through an optical image stabilization model to obtain a target image.

[0105] In this embodiment, by performing virtual optical image stabilization compensation through the optical image stabilization model when the focal length of the target focal length corresponding to the target zoom ratio selected by the user is within the preset focal length range, the application scenario range of the optical image stabilization compensation can be increased, and the clarity of the image captured at the focal length within the preset focal length range can be improved.

[0106] In an exemplary embodiment, Fig.10 As shown, the training method of the optical image stabilization model includes steps 1002 to 1006.

[0107] Step 1002, obtaining positive sample image data, negative sample image data and motion data samples corresponding to the motion trajectory under the same motion trajectory; the clarity of the positive sample image data is higher than the clarity of the negative sample image data.

[0108] Among them, the motion trajectory refers to the shaking trajectory of the lens. Positive sample image data and negative sample image data are obtained under the same motion trajectory, and the clarity of the positive sample image data is higher than that of the negative sample image data. It is easy to understand that the higher the clarity of the positive sample image data, the higher the quality of the optical image stabilization model finally trained. The positive sample image data and the negative sample image data under the same motion trajectory can form an image data sample pair, and the corresponding image data sample pairs can be obtained under multiple motion trajectories to train the optical image stabilization model. The more types of motion trajectories, the more accurate the trained optical image stabilization model will be. One motion trajectory corresponds to one motion data sample, and different motion trajectories usually correspond to different motion data samples.

[0109] In an optional example, if the camera module of the electronic device does not include an OIS component, image data collected by the electronic device under different motion trajectories can be used as negative sample image data, and motion blur removal processing can be performed on the negative sample image data to obtain corresponding clear image data, and the clear image data can be used as positive sample image data.

[0110] In another optional example, the camera module of the electronic device includes an OIS component, and the positive sample image data is acquired by enabling OIS and acquiring negative sample image data by disabling OIS while maintaining the same motion trajectory and corresponding to the same scene.

[0111] Step 1004: input the negative sample image data and the motion data sample into the initial anti-shake model to obtain predicted image data.

[0112] The initial anti-shake model may be any deblurring processing model, such as a convolutional neural network (CNN), a transformer model (Transformers) or a generative adversarial network (GAN), etc. The predicted image data refers to image data obtained by performing deblurring processing on the negative sample image data by the initial anti-shake model.

[0113] In an example, the motion data samples can be motion predicted by the initial anti-shake model to obtain motion information corresponding to the motion data samples, and the motion information corresponding to the motion data samples can be aligned with the standard motion information in the standard state to obtain the aligned motion information; blur prediction can be performed on the negative sample image data to obtain a deblurring kernel, and the negative sample image data can be deblurred based on the deblurring kernel to obtain deblurred image data; the deblurred image data and the aligned motion information can be reconstructed to obtain predicted image data.

[0114] Step 1006: training the initial anti-shake model according to the clarity difference between the predicted image data and the positive sample image data to obtain an optical anti-shake model.

[0115] The definition difference refers to the difference between the definition of the predicted image data and the definition of the positive sample image data. According to the definition difference, the model parameters of the initial anti-shake model can be adjusted in the direction of the definition close to the definition of the positive sample image data until the definition difference meets the difference threshold condition, and the optical anti-shake model is obtained. The difference threshold condition includes, for example, less than or equal to the difference threshold. Alternatively, when the actual training number reaches the preset training number, the optical anti-shake model is obtained.

[0116] In this embodiment, the initial anti-shake model is trained by using the positive sample image data and the negative sample image data under the same motion trajectory and the motion data samples corresponding to the motion trajectory to obtain the optical anti-shake model, so that the optical anti-shake model can learn to perform jitter compensation processing on the blurred image data according to the motion data corresponding to the motion trajectory, stabilize the object in the image, and obtain a clear image. Therefore, the optical anti-shake model can perform jitter compensation processing on the blurred image to obtain a clear image with motion blur suppressed, and there is no need to perform anti-shake compensation through hardware entities such as OIS components, which can reduce hardware costs while achieving anti-shake compensation. In addition, by adding motion data samples corresponding to the input motion trajectory to participate in the training of the optical anti-shake model, that is, adding more parameters to guide the model to learn the ability of jitter processing, the convergence speed of model training and the accuracy of the optical anti-shake model can be improved.

[0117] In some embodiments, step 1002 of acquiring positive sample image data, negative sample image data and motion data samples corresponding to the motion trajectory under the same motion trajectory includes:

[0118] Under the same motion trajectory, the optical image stabilization component is enabled to obtain positive sample image data, and the optical image stabilization component is disabled to obtain negative sample image data; and motion data samples corresponding to the motion trajectory are obtained.

[0119] In actual application scenarios, the optical image stabilization component can be enabled under the target motion trajectory to obtain positive sample image data. Similarly, the optical image stabilization component can be disabled under the target motion trajectory to obtain negative sample image data, and then the motion data sample corresponding to the target motion trajectory can be obtained.

[0120] In one example, if Fig.11As shown, the electronic device includes an OIS component. Under motion trajectory A, the OIS is controlled to be enabled to obtain positive sample image data A, and the OIS is controlled to be disabled to obtain negative sample image data A; under motion trajectory B, the OIS is controlled to be enabled to obtain positive sample image data B, and the OIS is controlled to be disabled to obtain negative sample image data B; under motion trajectory C, the OIS is controlled to be enabled to obtain positive sample image data C, and the OIS is controlled to be disabled to obtain negative sample image data C, etc. The types of motion trajectories may include N types, where N is a positive integer. Then, the initial anti-shake model is trained by the positive sample image data and negative sample image data obtained under all motion trajectories and the motion data samples corresponding to the motion trajectories to obtain an optical anti-shake model. Each time the initial anti-shake model is trained, the positive sample image data and negative sample image data under the same motion trajectory and the motion data samples corresponding to the motion trajectory are used. The initial anti-shake model can be trained in turn by the positive sample image data, negative sample image data and motion data samples corresponding to each type of motion trajectory.

[0121] In this embodiment, by respectively controlling the enabling and disabling of the optical image stabilization component under the same motion trajectory, positive sample image data and negative sample image data are correspondingly obtained. Since the positive sample image data is image data acquired when the optical image stabilization component is enabled, the positive sample image data has higher clarity, better motion blur suppression effect, and objects in the image are relatively stable. Therefore, in the process of training the optical image stabilization model, the model can fully learn the ability to stabilize objects and suppress motion blur in the positive sample image data. The trained optical image stabilization model can process images with motion blur into clear images with relatively stable objects. In addition, the training data samples of the optical image stabilization model can be easily and conveniently acquired, thereby reducing the cost of acquiring the training data samples and improving the efficiency of acquiring the training data samples.

[0122] In some embodiments, the step 1004 of inputting the negative sample image data and the motion data sample into the initial anti-shake model to obtain the predicted image data includes:

[0123] Based on the motion data samples, the anti-shake compensation amount corresponding to the negative sample image data is determined; the negative sample image data and the anti-shake compensation amount corresponding to the negative sample image data are input into the initial anti-shake model to obtain predicted image data.

[0124] It is easy to understand that based on the motion data sample, the anti-shake compensation amount corresponding to the negative sample image data is determined. Please refer to the description of determining the anti-shake compensation amount corresponding to the initial image data based on the motion data in the aforementioned embodiment. It is only necessary to replace the motion data sample with the motion data, which will not be repeated here.

[0125] In this embodiment, after obtaining the motion data sample, the anti-shake compensation amount corresponding to the negative sample image data is determined based on the motion data sample, and then the negative sample data and the anti-shake compensation amount corresponding to the negative sample data are input into the initial anti-shake model for training. This can reduce the calculation amount of the initial anti-shake model, improve the operation speed of the initial anti-shake model, and thus improve the model convergence speed.

[0126] In an exemplary embodiment, the process of the image processing method includes: obtaining initial image data without optical image stabilization components for image stabilization compensation, and motion data corresponding to the initial image data, the motion data including angular velocity data and acceleration data, the angular velocity data can be collected by a gyroscope, and the acceleration data can be collected by an accelerometer, and then according to the motion data, the image stabilization compensation amount target hall of the initial image data is determined, and the image stabilization compensation amount target hall includes the image stabilization compensation amount target hall X in the horizontal direction and the image stabilization compensation amount target hall Y in the vertical direction. The initial image data and the image stabilization compensation amount are input into the optical image stabilization model, and a target image compensated by virtual OIS, i.e., a virtual OIS compensated frame, can be obtained. Among them, the optical image stabilization model is equivalent to an AI (Artificial Intelligence) virtual motor, that is, the initial image data is subjected to jitter compensation processing by the optical image stabilization model.

[0127] The training process of the optical image stabilization model includes: obtaining image data training samples through a camera module including an OIS component, and controlling OIS to enable to obtain corresponding positive sample image data under each motion trajectory, such as obtaining positive sample image data A under motion trajectory A, obtaining positive sample data B under motion trajectory B, etc.; controlling OIS to disable to obtain corresponding negative sample image data under each motion trajectory in the same manner as the positive sample image data, such as obtaining negative sample image data A under motion trajectory A, obtaining negative sample data B under motion trajectory B, etc.; and obtaining gyroscope data that can express motion posture corresponding to each motion trajectory when OIS is enabled or disabled, that is, the motion data of the corresponding motion trajectory can be obtained in the process of obtaining the positive sample image data, or the motion data of the corresponding motion trajectory can be obtained in the process of obtaining the negative sample image data, and then the corresponding anti-shake compensation amount is calculated based on the gyroscope data. The negative sample image data and the corresponding anti-shake compensation amount are input into the initial anti-shake model to obtain the predicted image data processed by the initial anti-shake model. The initial anti-shake model is trained according to the difference in clarity between the positive sample image data and the predicted image data until the difference in clarity meets the difference threshold condition, thereby obtaining the optical anti-shake model. In other words, based on the collected positive sample image data and negative sample image data and the anti-shake compensation amount of the corresponding motion trajectory, the initial anti-shake model is trained to convergence. The convergence goal can be to make the clarity of the negative sample image data reach the clarity of the positive sample image data, that is, to enable the model to have the ability to process the blurred negative sample image data into a clear image identical to the positive sample image data, and to stabilize the object in the image data.

[0128] In actual application scenarios, the optical image stabilization model can be trained offline based on the positive sample image data and the negative sample image data obtained by switching OIS. When processing online, the video frames or image frames without OIS compensation and the corresponding anti-shake compensation amount are input to obtain the video frames or image frames with OIS motion blur suppression effect.

[0129] In the above embodiment, by acquiring image data sample pairs and motion data under corresponding motion trajectories and training the optical image stabilization model, it is possible to obtain image frames with OIS motion blur suppressed through the optical image stabilization model, which can avoid installing physical image stabilization components such as drive components in the camera module, reduce hardware costs and hardware debugging costs, and at the same time free up space for the camera module, reduce the volume of the camera module or the entire electronic device, and make the electronic device more miniaturized; in addition, OIS motion blur suppression based on the trained optical image stabilization model can achieve jitter compensation at a larger angle without being limited by hardware conditions, thereby achieving a more accurate motion blur suppression effect and being suitable for a wider range of shooting scenarios.

[0130] Fig.12 Schematic diagram of model selection interface 1200 in some exemplary embodiments. In some embodiments, model selection interface 1200 may be presented in a web browser. For example, a user may enter a website and present model selection interface 1200 to the user in a web browser of the website. As depicted, model selection interface 1200 may provide a variety of menus 1202-1210, options within these menus, and sub-options or categories 1212-1214, which users may select in order to implement the AI ​​functionality required for their application. It should be noted that the user interface shown here is for exemplary purposes and is not limited in any way to the menus, options, and sub-options shown herein. A variety of other menus, options, and sub-options are possible and within the scope of the present application.

[0131] like Fig.12 As shown, the model selection interface 1200 may include a task menu 1202, a device type menu 1204, and a constraint menu 1206-1210. The task menu 1202 may include a variety of AI tasks, and the user may select the desired task from these AI tasks to be incorporated into the user's application. By way of example and not limitation, these tasks may include scene recognition, image tagging, object detection, object tracking, object segmentation, human posture estimation, image enhancement, behavior recognition, human emotion recognition, speech recognition, text recognition, natural language understanding and / or any kind of data processing tasks. In a possible implementation scheme, one or more categories 1212 may be included in one or more of these tasks. The one or more categories within the task may include specific objects or items of interest that the user may wish to pay special attention to. For example, the categories within the object detection task may include people, trees, cars, bicycles, dogs, buildings, etc. For another example, the categories within the scene recognition task may include beaches, snow-capped mountains, flowers, buildings, autumn leaves, waterfalls, night scenes, etc. For another example, the categories within the image enhancement task may include denoising, super-resolution, etc.

[0132] The device type menu 1204 may include different types of user devices on which a given task (e.g., the task selected from the task menu 1202) may be deployed. By way of example and not limitation, user devices may include smartphones, tablets, smart watches, ARM-based platforms, embedded sensors, cameras, Intel-based platforms, drones, snapdragon-based platforms, NVIDIA-based platforms (e.g., various GPUs), X-86 series, ambarella platforms, etc. From the list of user devices, the user may select the device on which they wish to deploy the selected task.

[0133] Constraint menus 1206-1210 may include a memory constraint menu 1206, a delay constraint menu 1208, and a power constraint menu 1210. The memory constraint menu 1206 may enable a user to specify the amount of memory to be used for the task selected by the user. For example, as shown in the figure, the user may choose to allocate 100MB of memory to the object detection task. The delay constraint menu 1208 may enable a user to specify the refresh rate or frames per second (FPS) at which they wish their task to run. For example, as shown in the figure, the user may choose to run the object detection task at 10FPS. The power constraint menu 1210 may enable a user to specify the amount of power to be utilized by the task selected by the user. For example, as shown in the figure, the user may choose to allocate 1JPS (joules / second) of power to the object detection task.

[0134] Fig.13A 13 is a schematic diagram of various components and their associated functions of a model management system 1300 for assigning an appropriate AI model based on user specifications in some exemplary embodiments. Fig.12 The interface discussed in defines its specifications (e.g., tasks, device types, memory constraints, power constraints, latency constraints), and the user specifications can be sent to the query generator server 1302. The query generator server 1302 receives the user specifications and generates a query 1304, which can be used to retrieve one or more original models 1308 (or device-level models) from the database 1306. The device-level model can be defined in certain programming languages. For example, the device-level model can be defined in Python. The database 1306 may include multiple AI models that can be defined and stored based on previous user specifications. The previous user specification may be related to the current user specification or not. For example, the previous specification may be related to an image labeling task for an application intended to operate on an Android phone, and the model management system 1300 generates a model according to the specification and stores the model in the database 1306 for future access and / or retrieval. It is advantageous to store previous models or have a database of models because this avoids creating models from scratch, and existing models can be reused and adjusted to match current user specifications.

[0135] The original or high-level model 1308 retrieved from the database 1306 may be processed by a model compiler 1310 to make it appropriate according to the current user specifications. For example, the model compiler 1310 may compile the model to remove any unnecessary modules / components and add any required components to match the user specifications. For example, the original model may be pre-generated based on user specifications such as task: object detection; device type: Android; memory constraint: 1GB; latency constraint: 100FPS; power constraint: 1JPS; and some additional constraints. The model compiler 1310 may compile the original model to remove the additional constraints, and may change the device type to a mobile phone and the memory constraint to 100MB to make the model suitable for the current user specifications. In a possible embodiment, the output of the model compiler 1310 is a compact model package or model binary 1312. In a possible embodiment, compiling the original model essentially converts the high-level or original model into a model binary, which may also include user queries, such as Fig.13A shown.

[0136] The interface binder 1314 may bind the compact model / model binary to one or more interface functions. The interface function may be a function that actual users will use in an application or software once the model is deployed into the application. For example, to perform an image labeling task, the interface function may process an input image, and the output of the function will be a bunch of labels within the image.

[0137] Once one or more interface functions are bound to the model binary, the final software binary (a combination of the model binary and the interface functions) can be sent to a distributor 1318, which creates a link 1320 for the user to download the final software binary 1316 on their device. Upon download, the user can incorporate the received binary into their application, which can then be ultimately released for use by the end user. The end user can use the developer's software to perform the AI ​​functions.

[0138] In a possible implementation, an automatic benchmarking component built into the model can automatically evaluate the performance of the model based on, for example, how well the provided model serves the needs of the user and various performance constraints associated with the model (e.g., amount of memory and power usage, speed, etc.). The automatic benchmarking component can report the data obtained from the benchmarking to the model management system 1300, which can store the data in the database 1306 of AI models and use it to train the model. In some implementations, model training can also be performed based on feedback provided by the user at query time.

[0139] Fig. 13BSchematic diagram of additional components and associated functions of the model management system 1300 for training an AI model pool in some exemplary embodiments. The training of a model may include improving the performance of an existing model, adding one or more new features to an existing model, and / or creating a new model. A user may provide training data at query time, or in other words, when a user defines their specifications using the model selection interface 1200. By way of example and not limitation, the training data may include a request from a user to perform object detection for an object that may not currently be included in the model selection interface 1200, but the user may have data corresponding to those objects. The user may then provide their data along with their desired specifications to the model management system 1300, which may use the data to refine its existing models related to object detection to include new object categories provided by the user. For another example, a user may provide performance feedback for the model at query time (e.g., the extent to which the previous model serves its needs in terms of speed, memory, and power usage), and the model management system 1300 may use the performance feedback to train the AI ​​model pool on the server (indicated by reference numeral 1322). Other possible ways of training the model are possible and within the scope of this disclosure.

[0140] The trained AI models 1324 may be sent to a ranker 1326 for ranking. The ranker 1326 may rank the models based on various factors. These factors may include, for example, performance metrics associated with each of the models, benchmark data, user feedback on the models, and the like. Based on the ranking, the highest ranking models 1328 may be determined. For example, the top five ranked models may be selected and sent to the model converter 1330 of the device. The model converter 1330 may convert the highest ranked models into device-level models 1332, as discussed above. Automatic benchmarking may be performed on these models in terms of speed, memory, and power constraints (indicated by reference numeral 1334), and finally these device-level models may be stored in a database of AI models 1306 along with the benchmark data for future access and / or retrieval. In a possible embodiment, Fig. 13B Steps 1322-1334 shown may be used with Fig.13A The steps shown are performed asynchronously.

[0141] Fig.14 Flowchart of an embodiment of a testing process for a neural network module in some exemplary embodiments. During the testing process, a sample image input may be provided to the neural network module along with operating parameters. The neural network module may provide sample output data by processing the sample input image using the operating parameters. The sample output data may be compared to known data for the sample image to see if the data is matched in the matching data.

[0142] If the sample output data matches the sample image known data, then the operating parameters are set. If the sample output data does not match the sample image known data (within a desired tolerance), then the training process can be fine-tuned. Fine-tuning the training process can include providing additional training images to the training process and / or other adjustments in the training process to optimize the operating parameters of the neural network module (or generate new operating parameters).

[0143] Once the operating parameters for the neural network module are set in setting the operating parameters, the operating parameters may be applied to the device by providing the operating parameters to the neural network module in the electronic device or server. In certain embodiments, the operating parameters of the neural network module used for training are in a different numerical representation mode than the operating parameters of the neural network module applied on the device. For example, the neural network module used for training may use floating point numbers, while the neural network module used on the device uses integers. Therefore, in such embodiments, the operating parameters of the neural network module used for training are converted from floating point operating parameters to integer operating parameters of the neural network module applied on the device.

[0144] After providing the operating parameters to the neural network module on the device, the neural network module can operate on the device to implement the face detection process on the device. Fig.15 A flowchart of a neural network module on a device implementing a facial detection process in some exemplary embodiments. The image input may include an image captured using a camera on the device. The captured image may be a flood infrared illuminated image or a depth map image. The facial detection process may be used to detect whether a face is present in the image (e.g., placing a bounding box around the face), and if a face is detected, then evaluating values ​​of attributes of the face (e.g., position, pose, and / or distance).

[0145] The captured image from the image input can be provided to the encoder process. The encoder process can be performed by an encoder. In certain embodiments, the encoder module is a multi-scale convolutional neural network. In the encoder process, the encoder module can encode the image input to represent the features in the image as feature vectors in a feature space. The encoder process can output a feature vector. The feature vector can be, for example, an encoded image feature represented as a vector.

[0146] The feature vector may be provided to a decoder process. The decoder process may be performed by an encoder module. In certain embodiments, the decoder module may be a repetitive neural network. In the decoder process, the decoder module may decode the feature vector to evaluate one or more attributes of the image input, thereby determining (e.g., extracting) output data from the image input. Decoding the feature vector may include classifying the feature vector using classification parameters determined during the training process. Classifying the feature vector may include operating the feature vector using one or more classifiers or networks that support classification.

[0147] In some embodiments, the decoder process includes decoding the feature vectors for each region in the feature space. The feature vectors from each region of the feature space can be decoded into non-overlapping boxes in the output data. In some embodiments, decoding the feature vectors of a region (e.g., extracting information from the feature vectors) includes determining (e.g., detecting) whether there is a face in the region. Since the decoder process operates on each region in the feature space, the decoder module can provide a face detection score for each region in the feature space (e.g., based on a prediction of a confidence score as to whether a face or a part of a face is detected / existed in the region). In some embodiments, using an RNN, multiple predictions about whether there is a face (or a part of a face) can be provided for each region of the feature space, wherein the prediction includes predictions about both the face part within the region and the face part around the region (e.g., in an adjacent region). These predictions can be folded into a final decision on the presence of a face in the image input (e.g., detection of a face in the image input). In some embodiments, the prediction is used to form (e.g., place) a bounding box around a face detected in the image input. The output data may include a decision on the presence of a face in the image input (e.g., in a captured image) and a bounding box formed around the face.

[0148] In some embodiments, a face detection process detects the presence of a face in an image input regardless of the orientation of the face in the image input. For example, a neural network module implemented in an electronic device may operate a face detection process using operating parameters implemented by a training process developed to detect faces in any orientation in an image input, as described above. Thus, the face detection process may detect faces in an image input having any orientation in the image input without rotating the image and / or receiving any other sensor input data (e.g., accelerometer or gyroscope data) that may provide information about the orientation of the image. Detecting faces in any orientation also increases the range of pose estimates in a bounding box. For example, roll estimates in a bounding box are from -180° to +180° (all roll orientations).

[0149] In some embodiments, a face detection process detects the presence of partial faces in an image input. Partial faces detected in an image input may include any portion of a face present in the image input. The amount of faces that need to be present in the image input to be detected and / or the ability of the neural network module to detect partial faces in the image input may depend on the operating parameters used to generate the neural network module and / or the training of facial features detectable in the image input.

[0150] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0151] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0152] In an exemplary embodiment, Fig.16 As shown, an image processing device is provided, including: a data acquisition module 1602, an anti-shake compensation amount determination module 1604 and an anti-shake compensation module 1606, wherein:

[0153] A data acquisition module 1602 is used to acquire initial image data and motion data corresponding to the initial image data;

[0154] The anti-shake compensation amount determination module 1604 is used to determine the anti-shake compensation amount corresponding to the initial image data according to the motion data;

[0155] The anti-shake compensation module 1606 is used to input the initial image data and the anti-shake compensation amount into the optical anti-shake model to obtain a target image corresponding to the initial image data.

[0156] In some embodiments, the anti-shake compensation module 1606 is also used to perform motion prediction on the anti-shake compensation amount through an optical anti-shake model to obtain motion information; deblur the initial image data to obtain deblurred image data; and reconstruct the image based on the deblurred image data and motion information to obtain a target image.

[0157] In some embodiments, the optical image stabilization model includes a motion prediction network, a blur prediction network, an alignment network, a deblurring network, and a reconstruction network; the anti-shake compensation module 1606 is also used to perform motion prediction on the anti-shake compensation amount through the motion prediction network to obtain motion information; align the motion information with the standard motion information in the standard state through the alignment network to obtain aligned motion information; perform blur prediction on the initial image data through the blur prediction network to obtain a deblurring kernel; perform deblurring processing on the initial image data based on the deblurring kernel through the deblurring network to obtain deblurred image data; perform image reconstruction on the deblurred image data and the aligned motion information through the reconstruction network to obtain a target image.

[0158] In some embodiments, the motion data includes angular velocity data and acceleration data; the anti-shake compensation amount determination module 1604 is also used to determine the shake angle according to the angular velocity data; determine the shake displacement according to the acceleration data; and determine the anti-shake compensation amount corresponding to the initial image data based on the shake angle and the shake displacement.

[0159] In some embodiments, the data acquisition module 1602 is also used to, if there is a moment in the collection time of the motion data that is the same as the frame moment of the initial image data, use the motion data corresponding to the frame moment as the motion data corresponding to the initial image; if there is no moment in the collection time of the motion data that is the same as the frame moment of the initial image data, then among the collection times of the motion data, select at least two collection times closest to the frame moment of the initial image data; and determine the motion data corresponding to the frame moment of the initial image data based on the at least two collection times and the motion data corresponding to the at least two collection times.

[0160] In some embodiments, the initial image data includes target video frame data; the data acquisition module 902 is further used to acquire the target video frame data in the initial video stream and the motion data corresponding to the target video frame data;

[0161] The above-mentioned image processing device also includes a target video stream generation module, which is used to obtain a target video stream corresponding to the initial video stream based on the target images corresponding to all target video frame data; or, to obtain a target video stream corresponding to the initial video stream based on the video frame data other than the target video frame data in the initial video stream and the target images corresponding to all target video frame data.

[0162] In some embodiments, the data acquisition module 1602 is further configured to perform optical image stabilization compensation through an optical image stabilization component to obtain initial image data.

[0163] In some embodiments, the above-mentioned image processing device also includes a focal length recognition module, which is used to respond to the selection trigger of the target zoom ratio, determine the target focal length according to the target zoom ratio, and if the focal length corresponding to the target focal length is within a preset focal length range, the initial image data and the anti-shake compensation amount are input into the optical anti-shake model to obtain the target image corresponding to the initial image data.

[0164] In some embodiments, the above-mentioned image processing device also includes a model training module, which is used to obtain positive sample image data, negative sample image data and motion data samples corresponding to the motion trajectory under the same motion trajectory; the clarity of the positive sample image data is higher than the clarity of the negative sample image data; the negative sample image data and the motion data samples are input into the initial anti-shake model to obtain predicted image data; according to the clarity difference between the predicted image data and the positive sample image data, the initial anti-shake model is trained to obtain the optical anti-shake model.

[0165] In some embodiments, the model training module is also used to control the enabling of the optical image stabilization component to obtain positive sample image data, and to control the disabling of the optical image stabilization component to obtain negative sample image data under the same motion trajectory; and to obtain motion data samples corresponding to the motion trajectory.

[0166] In some embodiments, the model training module is also used to determine the anti-shake compensation amount corresponding to the negative sample image data based on the motion data sample; the negative sample image data and the anti-shake compensation amount corresponding to the negative sample image data are input into the initial anti-shake model to obtain predicted image data.

[0167] Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0168] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.17As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0169] Those skilled in the art will understand that Fig.17 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0170] In an exemplary embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the image processing method in the above embodiment when executing the computer program.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method in the above embodiment are implemented.

[0172] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the image processing method in the above embodiment when executed by a processor.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0174] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0175] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0176] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire initial image data and motion data corresponding to the initial image data; determining an anti-shake compensation amount corresponding to the initial image data according to the motion data; The initial image data and the anti-shake compensation amount are input into an optical anti-shake model to obtain a target image corresponding to the initial image data.

2. The method according to claim 1, characterized in that The step of inputting the initial image data and the anti-shake compensation amount into an optical anti-shake model to obtain a target image corresponding to the initial image data includes: Performing motion prediction on the anti-shake compensation amount by using the optical anti-shake model to obtain motion information; Performing a deblurring process on the initial image data to obtain deblurred image data; Image reconstruction is performed based on the deblurred image data and the motion information to obtain the target image.

3. The method according to claim 1, characterized in that The motion data includes angular velocity data and acceleration data; and determining the anti-shake compensation amount corresponding to the initial image data according to the motion data includes: Determining a shaking angle according to the angular velocity data; Determining a jitter displacement according to the acceleration data; Based on the shaking angle and the shaking displacement, an anti-shake compensation amount corresponding to the initial image data is determined.

4. The method according to claim 1, characterized in that: Acquiring motion data corresponding to the initial image data, including: If there is a time in the acquisition time of the motion data that is the same as the frame time of the initial image data, the motion data corresponding to the frame time is used as the motion data corresponding to the initial image; If there is no time in the acquisition time of the motion data that is the same as the frame time of the initial image data, then at least two acquisition times closest to the frame time of the initial image data are selected from the acquisition times of the motion data; The motion data corresponding to the frame time of the initial image data is determined according to the at least two acquisition moments and the motion data corresponding to the at least two acquisition moments.

5. The method according to claim 1, characterized in that The initial image data includes target video frame data; the acquiring of the initial image data and the motion data corresponding to the initial image data includes: Acquire target video frame data in an initial video stream and motion data corresponding to the target video frame data; The method further comprises: Obtaining a target video stream corresponding to the initial video stream according to the target images corresponding to all target video frame data; Alternatively, the target video stream corresponding to the initial video stream is obtained based on the video frame data other than the target video frame data in the initial video stream and the target images corresponding to all the target video frame data.

6. The method according to claim 1, characterized in that The obtaining of initial image data comprises: Initial image data obtained by performing optical image stabilization compensation through an optical image stabilization component is acquired.

7. The method according to claim 1, characterized in that The initial image data includes blurred image data.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: In response to the selection trigger of the target zoom ratio, the target focal length is determined according to the target zoom ratio. If the focal length corresponding to the target focal length is within a preset focal length range, the step of inputting the initial image data and the anti-shake compensation amount into the optical anti-shake model to obtain the target image corresponding to the initial image data is executed.

9. The method according to claim 1, characterized in that: The training method of the optical image stabilization model includes: Acquire positive sample image data, negative sample image data and motion data samples corresponding to the motion trajectory under the same motion trajectory; the clarity of the positive sample image data is higher than the clarity of the negative sample image data; Inputting the negative sample image data and the motion data sample into an initial anti-shake model to obtain predicted image data; The initial anti-shake model is trained according to the definition difference between the predicted image data and the positive sample image data to obtain the optical anti-shake model.

10. The method according to claim 9, characterized in that The obtaining of positive sample image data, negative sample image data and motion data samples corresponding to the motion trajectory under the same motion trajectory includes: Under the same motion trajectory, the optical image stabilization component is enabled to obtain positive sample image data, and the optical image stabilization component is disabled to obtain negative sample image data; Obtain motion data samples corresponding to the motion trajectory.

11. The method according to claim 9, characterized in that The step of inputting the negative sample image data and the motion data sample into an initial anti-shake model to obtain predicted image data includes: Based on the motion data sample, determining an anti-shake compensation amount corresponding to the negative sample image data; The negative sample image data and the anti-shake compensation amount corresponding to the negative sample image data are input into an initial anti-shake model to obtain the predicted image data.

12. An image processing device, characterized in that: The device comprises: A data acquisition module, used to acquire initial image data and motion data corresponding to the initial image data; an anti-shake compensation amount determination module, configured to determine the anti-shake compensation amount corresponding to the initial image data according to the motion data; The anti-shake compensation module is used to input the initial image data and the anti-shake compensation amount into an optical anti-shake model to obtain a target image corresponding to the initial image data.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images