Image processing method and apparatus therefor

Abstract

The present application belongs to the technical field of image processing, and discloses an image processing method and an apparatus therefor. The image processing method is executed by an electronic device. The image processing method comprises: acquiring first pose information of a first image and second pose information of a second image (S102); according to the first pose information and the second pose information, determining disturbance information between the first image and the second image (S104); according to the disturbance information, correcting the second pose information, and obtaining third pose information (S106); performing smoothing processing on the third pose information, and obtaining fourth pose information (S108); and, according to the fourth pose information and the second image, obtaining a third image (S110).

Description

Image processing method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on March 19, 2024, with application number 202410315357.2 and entitled “Image Processing Method and Device Thereof,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application belongs to the field of image processing technology, and specifically relates to an image processing method and device thereof. Background Art

[0004] Currently, it's becoming a trend for electronic devices to be equipped with multiple cameras of varying focal lengths. Switching between cameras during zoom operations while shooting video can yield better image quality. However, due to the varying viewing angles and parameters of each camera, switching between cameras often results in noticeable image jitter, resulting in lower image quality.

[0005] To reduce image jitter during camera switching, the current traditional camera switching solution involves simultaneously activating multiple cameras and registering the image captured by the current camera with that captured by a pre-cut camera in real time. Feature points derived from the registration are then used to infer and crop the current image to achieve smooth multi-camera switching. However, activating multiple cameras simultaneously increases the power consumption of electronic devices, and the smoothing effect of this solution is limited. The image captured at the moment of camera switching still experiences significant jitter, reducing image quality. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide an image processing method and device thereof, which can solve the problem that the camera of an electronic device shoots a significant jump in the picture at the moment of switching, resulting in a decrease in the quality of the shot picture.

[0007] In a first aspect, an embodiment of the present application provides an image processing method, which is executed by an electronic device. The image processing method includes: obtaining first posture information of a first image and second posture information of a second image; determining disturbance information between the first image and the second image based on the first posture information and the second posture information; correcting the second posture information based on the disturbance information to obtain third posture information; smoothing the third posture information to obtain fourth posture information; and obtaining a third image based on the fourth posture information and the second image.

[0008] In a second aspect, an embodiment of the present application provides an image processing device, which is applied to an electronic device, and the image processing device includes: a processing unit, used to obtain first posture information of a first image and second posture information of a second image; the processing unit is also used to determine disturbance information between the first image and the second image based on the first posture information and the second posture information; the processing unit is also used to correct the second posture information based on the disturbance information to obtain third posture information; the processing unit is also used to smooth the third posture information to obtain fourth posture information; the processing unit is also used to obtain a third image based on the fourth posture information and the second image.

[0009] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the image processing method of the first aspect are implemented.

[0010] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a program or instruction stored thereon, which, when executed by a processor, implements the steps of the image processing method of the first aspect.

[0011] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the image processing method of the first aspect.

[0012] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the image processing method of the first aspect.

[0013] In the image processing method provided in the embodiment of the present application, first posture information of a first image and second posture information of a second image are obtained; disturbance information between the first image and the second image is determined based on the first posture information and the second posture information; the second posture information is corrected based on the disturbance information to obtain third posture information; the third posture information is smoothed to obtain fourth posture information; and the third image is obtained based on the fourth posture information and the second image. Through the above-mentioned image processing method, in the process of switching cameras for shooting, multiple cameras are regarded as a whole camera, and the switching of different cameras is regarded as inter-frame disturbance. Based on the posture information and disturbance information of different frame images, the picture shot after the camera is switched is processed, thereby processing the inter-frame disturbance caused by the camera switching. In this way, the problem that the camera of the electronic device has a significant jump in the picture shot at the moment of switching, thereby reducing the quality of the shot picture, is solved, and smooth switching of the camera is achieved. Moreover, during the camera switching process, only a single camera needs to be kept turned on at the same time, which reduces the power consumption of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a flow chart of an image processing method according to an embodiment of the present invention;

[0015] FIG2 is a second flow chart of the image processing method provided in an embodiment of the present application;

[0016] FIG3 is a third flow chart of the image processing method provided in an embodiment of the present application;

[0017] FIG4 is a fourth flow chart of the image processing method provided in an embodiment of the present application;

[0018] FIG5 is a fifth flow chart of the image processing method provided in an embodiment of the present application;

[0019] FIG6 is a sixth flow chart of the image processing method provided in an embodiment of the present application;

[0020] FIG7 is a schematic diagram of an image processing method according to an embodiment of the present application;

[0021] FIG8 is a second schematic diagram of the image processing method provided in an embodiment of the present application;

[0022] FIG9 is a structural block diagram of an image processing device provided in an embodiment of the present application;

[0023] FIG10 is a structural block diagram of an electronic device provided in an embodiment of the present application;

[0024] FIG11 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. Specific embodiments

[0025] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field fall within the scope of protection of this application.

[0026] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0028] As shown in FIG1 , an embodiment of the present application provides an image processing method, which may include the following steps S102 to S110 :

[0029] S102: Acquire first posture information of the first image and second posture information of the second image.

[0030] The image processing method proposed in the embodiment of the present application is executed by an electronic device, which may specifically be an intelligent electronic device such as a smartphone, a tablet computer, a laptop computer, and a smart watch, and is not specifically limited here.

[0031] Among them, the first image and the second image correspond to different cameras in the camera module of the electronic device. For example, the first image is the image taken by the first camera in the camera module, and the second image is the image taken by the second camera in the camera module. No specific restrictions are given here.

[0032] Furthermore, taking the example of the first image being captured by the first camera and the second image being captured by the second camera, the first image is the last frame image captured by the first camera during the process of switching from the first camera to the second camera, and the second image is the first frame image captured by the second camera during the process of switching from the first camera to the second camera, that is, the switching frame image.

[0033] Furthermore, the first posture information is real-time posture information of the first image, and the second posture information is real-time posture information of the second image. In actual application, the first posture information and the second posture information are both expressed in the form of a matrix.

[0034] Furthermore, in the image processing method provided in the embodiments of the present application, the multiple cameras in the camera module are considered as a whole, and the real-time posture information of the camera module is used as the real-time posture information of the captured image. That is, the first posture information may specifically be the real-time posture information of the camera module when capturing the first image, and the second posture information may specifically be the real-time posture information of the camera module when capturing the second image.

[0035] It is understood that shaking of an electronic device can cause changes in the real-time posture information of the camera module, and therefore in the real-time posture information of the captured image. Based on this, the first and second posture information can be used to reflect the shaking of the electronic device itself during camera switching, reflecting the posture changes of the captured image during camera switching.

[0036] S104: Determine disturbance information between the first image and the second image according to the first posture information and the second posture information.

[0037] The disturbance information is used to reflect the disturbance transformation of the captured image during camera switching.

[0038] Specifically, in the image processing method provided in the embodiment of the present application, in the process of switching cameras for shooting, the multiple cameras in the camera module are regarded as a whole camera, and the switching of different cameras is regarded as inter-frame disturbance. Based on the real-time posture information of the first image and the second image, the inter-frame disturbance caused by the camera switching is modeled to obtain the above-mentioned disturbance information.

[0039] S106: Correct the second posture information according to the disturbance information to obtain third posture information.

[0040] The second posture information is the real-time posture information of the second image, that is, the second posture information is the initial posture information of the second image.

[0041] It is understandable that, during the camera switching process, the second image captured after the camera is switched has not only a posture change caused by the shaking of the electronic device itself, but also a disturbance change caused by the camera switching, compared to the first image captured before the camera is switched. Therefore, in the image processing method provided in the embodiment of the present application, during the process of switching the camera for shooting, based on the real-time posture information of the first image and the second image, the disturbance information between the first image and the second image during the camera switching process is determined, and then the initial posture information of the second image, i.e., the second posture information, is corrected according to the obtained disturbance information to obtain the third posture information, so as to reduce the posture change and disturbance change between the first image and the second image during the camera switching process.

[0042] S108: Smoothing the third posture information to obtain fourth posture information.

[0043] Among them, the above-mentioned smoothing processing can specifically be a smoothing processing method based on a Kalman filter, or a smoothing processing method based on deep learning, and no specific limitation is made here.

[0044] The Kalman filter is a common state estimation method that estimates system states by filtering observed data. Deep learning-based smoothing methods, on the other hand, use neural networks to learn the characteristics of disturbances and jitters, thereby achieving smoothing.

[0045] Specifically, in the image processing method provided in the embodiments of the present application, after the real-time posture information of the second image is corrected to obtain the third posture information, the third posture information of the second image is smoothed using a Kalman filter or a deep learning-based method based on the electronic device anti-shake algorithm. In this way, the inter-frame disturbance caused by camera switching and the inherent jitter of the electronic device are modeled, and the above disturbance and jitter are processed using a smoothing method based on the anti-shake principle, thereby achieving smooth switching between multiple cameras.

[0046] In actual application, a sliding window method can be used to select a certain number of consecutive frames, i.e., captured images. For example, starting from the first frame of the image being captured, multiple frames of the image are selected for smoothing, or starting from a few frames of the image captured before the camera is switched, multiple frames of the image are selected for smoothing. The number of image frames selected and the sliding window interval can depend on the number and performance of the caches configured in the electronic device. The more image frames selected, the better the smoothing effect. Furthermore, for images captured before the camera is switched, such as the first image, the real-time posture information of the captured image is smoothed, and for images captured after the camera is switched, such as the second image, the real-time posture information of the corrected captured image is smoothed.

[0047] For example, as shown in FIG8 , based on an electronic device anti-shake algorithm, a Kalman filter or a deep learning-based method is used to smooth the i-1th and i-th frames based on the real-time posture information of the i-1th and i-th frames captured before the camera is switched. Furthermore, based on the corrected real-time posture information of the i+1th and i+2th frames captured after the camera is switched, the i+1th and i+2th frames are smoothed. As shown in FIG8 , the video stream smoothed by the algorithm can achieve smooth and stable switching between multiple cameras.

[0048] S110: Obtain a third image according to the fourth posture information and the second image.

[0049] Specifically, in the image processing method provided in the embodiment of the present application, after smoothing the third posture information of the second image to obtain the corresponding fourth posture information, the obtained fourth posture information is applied to the second image to obtain the corresponding third image. Compared with the second image, the posture transformation and disturbance transformation between the third image and the first image are smaller, thereby realizing smooth switching between multiple cameras and improving the quality of the captured image during the camera switching process.

[0050] The above-mentioned image processing method provided by the embodiment of the present application obtains the first posture information of the first image and the second posture information of the second image; determines the disturbance information between the first image and the second image based on the first posture information and the second posture information; corrects the second posture information based on the disturbance information to obtain the third posture information; smoothes the third posture information to obtain the fourth posture information; and obtains the third image based on the fourth posture information and the second image. Through the above-mentioned image processing method, in the process of switching the camera for shooting, the multiple cameras in the camera module are regarded as a whole camera, and the switching of different cameras is regarded as inter-frame disturbance. Based on the posture information and disturbance information of different frame images, the picture taken after the camera is switched is processed, thereby processing the inter-frame disturbance caused by the camera switching. In this way, the problem that the camera of the electronic device has a significant jump in the picture taken at the moment of switching, thereby reducing the quality of the captured picture, is solved, and the smooth switching of the camera is achieved. Moreover, during the camera switching process, only a single camera needs to be kept turned on at the same time, which reduces the power consumption of the electronic device.

[0051] In an embodiment of the present application, the electronic device includes an inertial measurement device, the first image and the second image correspond to different cameras in a camera module of the electronic device, and the first image is captured at a first moment. On this basis, as shown in FIG. 2 , the step of obtaining the first posture information of the first image may specifically include the following S112 to S116:

[0052] S112: Obtaining initial posture information of the camera module.

[0053] The above-mentioned initial posture information may specifically be a unit matrix, which is not specifically limited here.

[0054] S114: Determine a first rotation matrix according to output information of the inertial measurement device from the initial moment to the first moment and an output time interval of the inertial measurement device.

[0055] The inertial measurement device is a device for measuring the three-axis attitude angle and acceleration of an object. In actual application, the inertial measurement device can be an inertial measurement sensor, a gyroscope, or other device, which is not specifically limited here.

[0056] Furthermore, the first moment is the shooting moment of the first image, and the initial moment is the shooting start moment.

[0057] Furthermore, the output information is information such as the three-axis attitude angle and acceleration of the electronic device detected by the inertial measurement device.

[0058] Furthermore, the output time interval is the time interval between two adjacent output information output by the inertial measurement device.

[0059] Specifically, in the image processing method provided in the embodiment of the present application, based on the following formula (1), the real-time posture information of the camera module at the current moment is used to calculate the real-time posture information of the camera module at the next moment: t+Δt =R t e (ω(t)Δt) , (1)

[0060] Where t is the current time, Δt is the output time interval of the inertial measurement unit, R t is the real-time posture information of the camera module at time t, R t+Δt is the real-time attitude information of the camera module at time (t+Δt), ω(t) is the output information of the inertial measurement device at time t, e (ω(t)Δt) Represents the rotation matrix. By continuously repeating the above formula (1), the posture information of the camera module can be obtained in real time.

[0061] On this basis, by transforming the above formula (1), the corresponding relationship between the real-time posture information of the camera module at the first moment, that is, the above-mentioned first posture information and the initial posture information of the camera module can be obtained, that is, the following formula (2) can be obtained:

[0062] Among them, t is the first moment, R t is the real-time attitude information of the camera module at time t, i.e., the first attitude information mentioned above; R0 is the initial attitude information of the camera module; Δt is the output time interval of the inertial measurement device; i is any time between 0 and (t-1); ω(i) is the output information of the inertial measurement device at time i; This is the first rotation matrix mentioned above.

[0063] S116: Determine first posture information according to the first rotation matrix and the initial posture information.

[0064] Specifically, in the image processing method provided in the embodiment of the present application, after obtaining the initial posture information of the camera module and the above-mentioned first rotation matrix, based on the above-mentioned formula (2), the first posture information of the first image is obtained by multiplying the first rotation matrix and the initial posture information.

[0065] Similarly, in actual application, when obtaining the second posture information of the second image, the first posture information of the first image can be used to calculate the real-time posture information of the second image, i.e., the second posture information, based on the following formula (3): R t+1 =R t e (ω(t)Δt) , (3)

[0066] Where t is the first moment, Δt is the output time interval of the inertial measurement device, ω(t) is the output information of the inertial measurement device at moment t, R t is the first posture information, R t+1 is the second posture information.

[0067] In the above-mentioned embodiment provided by the present application, the electronic device includes an inertial measurement device, the first image and the second image correspond to different cameras in the camera module of the electronic device, the first image is captured at a first moment, and in the process of obtaining the first posture information of the first image, the initial posture information of the camera module is obtained; a first rotation matrix is ​​determined based on the output information of the inertial measurement device from the initial moment to the first moment and the output time interval of the inertial measurement device; and the first posture information is determined based on the first rotation matrix and the initial posture information. In this way, the real-time and accuracy of the determination of the first posture information is guaranteed, thereby improving the accuracy of subsequent image processing based on the first posture information.

[0068] In the embodiment of the present application, as shown in FIG3 , the above S104 may specifically include the following S104a to S104e:

[0069] S104a: Determine a transformation matrix according to the first posture information and the second posture information.

[0070] The first posture information is the real-time posture information of the first image, and the second posture information is the real-time posture information of the second image.

[0071] It is understandable that the second image captured after the camera is switched is not only subject to the posture transformation caused by the shaking of the electronic device itself, but also to the disturbance transformation caused by the camera switching, compared to the first image captured before the camera is switched. Therefore, in the image processing method provided in the embodiment of the present application, after obtaining the first posture information and the second posture information, based on the following formula (4), the transformation matrix required to align the first image and the second image is determined according to the first posture information and the second posture information: M = K·R t_inv ·R t+1 ·R r ·T r ·K inv , (4)

[0072] Among them, M is the transformation matrix, K is the camera intrinsic parameter matrix, K inv is the inverse matrix of the camera intrinsic parameter matrix, R r is the second rotation matrix in the perturbation information to be solved, T r is the translation matrix in the disturbance information to be solved, R t_inv is the inverse matrix of the first posture information, R t+1 is the second posture information, and represents the dot product of the matrix.

[0073] S104b: Extracting a plurality of first feature points from the first image and a plurality of second feature points from the second image.

[0074] The plurality of first feature points and the plurality of second feature points correspond one to one.

[0075] Specifically, in the image processing method provided in the embodiment of the present application, the least squares method can be used to solve R in the above formula (4): r and T r Among them, the least squares method is a mathematical optimization method used to find a set of parameters so that the error between a given model and actual observation data is minimized.

[0076] Specifically, in the image processing method provided in the embodiment of the present application, in the process of determining the disturbance information between the first image and the second image, that is, in solving R in the above formula (4), r and T r In the process of extracting multiple first feature points from the first image, a first feature point set P{p i}, where p i represents the i-th first feature point extracted. Further, a plurality of corresponding second feature points are extracted from the second image to obtain a second feature point set Q{q i}, where q i represents the extracted i-th second feature point, p i With q i Corresponding.

[0077] S104c: Constructing an error function corresponding to each pair of the first feature point and the second feature point based on the transformation matrix.

[0078] Specifically, in the image processing method provided in an embodiment of the present application, after extracting multiple first feature points from the first image and extracting corresponding multiple second feature points from the second image, for each pair of corresponding first feature points and second feature points, an error function between each pair of first feature points and second feature points is constructed based on the above-mentioned transformation matrix.

[0079] The error function between each pair of first feature points and second feature points is used to measure the transformation errors of the positions of the first feature point and the second feature point in the first image and the second image respectively.

[0080] In actual application, the error function may specifically be Euclidean distance, reprojection error, etc., which is not specifically limited here.

[0081] For example, taking the above error function as an example, the error function between each pair of first feature points and second feature points can be specifically expressed as the following formula (5): f1=p i Mq i , (5)

[0082] Among them, f1 represents the error function, M is the transformation matrix, and p i is the pixel position of the i-th first feature point in the first image, q i is the pixel position of the i-th second feature point in the second image.

[0083] S104d: Constructing a target optimization function based on multiple error functions.

[0084] Specifically, in the image processing method provided in the embodiment of the present application, after constructing the error function corresponding to each pair of first feature points and second feature points, the target optimization function is constructed by minimizing the sum of squares of multiple error functions.

[0085] For example, taking the reprojection error as an example, the target optimization function can be specifically expressed as the following formula (6): f2 = min∑|p i Mq i | 2 , (6)

[0086] Where f2 represents the target optimization function.

[0087] S104e: Solve the target optimization function and obtain disturbance information.

[0088] Specifically, in the image processing method provided in the embodiment of the present application, after constructing the above-mentioned target optimization function, a numerical optimization algorithm is used to solve the above-mentioned target optimization function to obtain the second rotation matrix and translation matrix in the above-mentioned transformation matrix. The second rotation matrix and translation matrix are the above-mentioned disturbance information.

[0089] In actual application, the numerical optimization algorithm may specifically be a gradient descent algorithm, a LM (Levenberg-Marquardt) algorithm, etc., which is not specifically limited here.

[0090] In the above-mentioned embodiment provided by the present application, in the process of determining the disturbance information between the first image and the second image based on the first posture information and the second posture information, a transformation matrix is ​​determined based on the first posture information and the second posture information; multiple first feature points in the first image and multiple second feature points in the second image are extracted, with the multiple first feature points and the multiple second feature points corresponding one to one; an error function corresponding to each pair of first feature points and second feature points is constructed based on the transformation matrix; a target optimization function is constructed based on the multiple error functions; and the target optimization function is solved to obtain the disturbance information. In this way, the accuracy of determining the disturbance information is ensured, thereby improving the accuracy of subsequent image processing based on the disturbance information.

[0091] In the embodiment of the present application, the disturbance information includes a second rotation matrix and a translation matrix. On this basis, as shown in FIG4 , the above S106 may specifically include the following S106a:

[0092] S106a: Determine third posture information according to the product of the second posture information, the second rotation matrix and the translation matrix.

[0093] The above-mentioned disturbance information includes a second rotation matrix and a translation matrix.

[0094] It is understandable that, during the camera switching process, the second image captured after the camera is switched has a posture change caused by the shaking of the electronic device itself, as compared to the first image captured before the camera is switched. Therefore, in the image processing method provided in the embodiment of the present application, after determining the disturbance information between the first image and the second image, the second posture information of the second image is corrected by the second rotation matrix and translation matrix in the disturbance information based on the following formula (7), thereby obtaining the third posture information: R' t+1 =R t+1 ·R r ·T r , (7)

[0095] Among them, R' t+1 is the third posture information, R r is the second rotation matrix, T r is the translation matrix, R t+1 is the second posture information, and represents the dot product of the matrix.

[0096] For example, as shown in FIG7 , for a single-channel video stream, in the process of switching from camera A to camera B, the real-time posture information of the i-1th frame image captured by camera A is R i-1 , the real-time posture information of the i-th frame image taken by camera A is R i , the real-time posture information of the first frame image taken by camera B, i.e. the switching frame image, is Ri+1 , and the corrected real-time posture information is R i+1 ·R r ·T r , the real-time posture information of the second frame image taken by camera B is R i+2 , and the corrected real-time posture information is R i+2 ·R r ·T r By correcting the real-time posture information of the image captured after the camera is switched, the posture transformation and disturbance transformation of the image captured during the camera switching process are reduced.

[0097] In the above-mentioned embodiment provided by the present application, the disturbance information includes a second rotation matrix and a translation matrix. When correcting the second posture information based on the disturbance information to obtain the third posture information, the third posture information is determined based on the product of the second posture information, the second rotation matrix, and the translation matrix. In this way, correcting the real-time posture information of the second image captured after the camera is switched can reduce the pose transformation and disturbance transformation between the first and second images during the camera switching process, thereby improving the quality of the image captured during the camera switching.

[0098] In the embodiment of the present application, as shown in FIG5 , after determining the disturbance information between the first image and the second image, the image processing method may further include the following steps S118 to S122:

[0099] S118: Determine a fourth image according to the first image and the disturbance information.

[0100] The above-mentioned disturbance information includes a second rotation matrix and a translation matrix.

[0101] Specifically, in the image processing method provided in an embodiment of the present application, after solving the above-mentioned target optimization function to obtain the disturbance information between the first image and the second image, the second rotation matrix and translation matrix in the determined disturbance information are applied to the above-mentioned first image to transform the first image to obtain the corresponding fourth image.

[0102] S120: Compare the second image and the fourth image, and determine the accuracy of the disturbance information according to the comparison result.

[0103] Specifically, in the image processing method provided in an embodiment of the present application, after obtaining the above-mentioned fourth image, the second image and the fourth image are compared to obtain the accuracy of the above-mentioned disturbance information based on the comparison result, that is, to obtain the accuracy of solving the above-mentioned target optimization function.

[0104] S122: When the accuracy is less than a preset threshold, adjust the disturbance information.

[0105] The preset threshold is used to determine whether the accuracy of the disturbance information meets the requirements. If the accuracy of the disturbance information is less than the preset threshold, it indicates that the accuracy of the disturbance information is low and needs to be re-solved.

[0106] Specifically, in the image processing method provided in the embodiment of the present application, after determining the accuracy of the disturbance information, the accuracy is compared with a preset threshold. If the accuracy is less than the preset threshold, it indicates that the accuracy of the disturbance information is low. At this time, the above-mentioned disturbance information is re-solved to adjust and optimize the disturbance information to ensure the accuracy of the obtained disturbance information.

[0107] Among them, the specific value of the above-mentioned preset threshold can be set by those skilled in the art according to actual conditions and is not specifically limited here.

[0108] In the above-described embodiment provided by this application, after determining the disturbance information between the first and second images, a fourth image is determined based on the first image and the disturbance information. The second and fourth images are compared, and the accuracy of the disturbance information is determined based on the comparison result. If the accuracy is less than a preset threshold, the disturbance information is adjusted. This ensures the accuracy of the obtained disturbance information, thereby improving the accuracy of subsequent image processing based on the disturbance information.

[0109] In summary, as shown in FIG6 , the image processing method provided in the embodiment of the present application may specifically include the following steps S202 to S208 :

[0110] S202: Record the posture information of each frame of image in real time.

[0111] S204: Calculating disturbance information between the switching frame image and the previous frame image when switching the camera.

[0112] S206: Correcting the real-time posture information of the switching frame image according to the disturbance information.

[0113] S208: Smoothing the corrected real-time posture information, and applying the smoothed real-time posture information to the switching frame image.

[0114] Among them, the previous frame image is the above-mentioned first image, the switching frame image is the above-mentioned second image, the real-time posture information of the switching frame image is the above-mentioned second posture information, the corrected real-time posture information is the above-mentioned third posture information, and the real-time posture information after smoothing is the above-mentioned fourth posture information.

[0115] Thus, the image processing method provided by the embodiment of the present application treats the multiple cameras in the camera module as a single camera, and treats the switching between different cameras as inter-frame disturbances. Based on the video stabilization method, this frame disturbance is smoothed, achieving the effect of smooth multi-camera switching. Furthermore, based on the image processing method provided by the embodiment of the present application, only a single camera needs to be kept active during the camera switching process, reducing the power consumption of the electronic device.

[0116] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the above-mentioned image processing method as an example.

[0117] As shown in FIG9 , an embodiment of the present application provides an image processing apparatus 500 , which is applied to an electronic device. The image processing apparatus 500 may include the following processing unit 502 .

[0118] The processing unit 502 is configured to obtain first posture information of the first image and second posture information of the second image;

[0119] The processing unit 502 is further configured to determine disturbance information between the first image and the second image based on the first posture information and the second posture information;

[0120] The processing unit 502 is further configured to correct the second posture information according to the disturbance information to obtain third posture information;

[0121] The processing unit 502 is further configured to perform smoothing on the third posture information to obtain fourth posture information;

[0122] The processing unit 502 is further configured to obtain a third image according to the fourth posture information and the second image.

[0123] The image processing device 500 provided in the embodiment of the present application obtains the first posture information of the first image and the second posture information of the second image; determines the disturbance information between the first image and the second image based on the first posture information and the second posture information; corrects the second posture information based on the disturbance information to obtain the third posture information; smoothes the third posture information to obtain the fourth posture information; and obtains the third image based on the fourth posture information and the second image. Through the above-mentioned image processing device 500, in the process of switching the camera for shooting, the multiple cameras in the camera module are regarded as a whole camera, and the switching of different cameras is regarded as inter-frame disturbance. Based on the posture information and disturbance information of different frame images, the picture taken after the camera is switched is processed, thereby processing the inter-frame disturbance caused by the camera switching. In this way, the problem that the camera of the electronic device has a significant jump in the picture taken at the moment of switching, thereby reducing the quality of the captured picture, and the smooth switching of the camera is achieved. In addition, during the camera switching process, only a single camera needs to be kept turned on at the same time, which reduces the power consumption of the electronic device.

[0124] In an embodiment of the present application, the electronic device includes an inertial measurement device, the first image and the second image correspond to different cameras in a camera module of the electronic device, the shooting time of the first image is a first moment, and the processing unit 502 is specifically used to: obtain initial posture information of the camera module; determine a first rotation matrix based on the output information of the inertial measurement device from the initial moment to the first moment and the output time interval of the inertial measurement device; determine the first posture information based on the first rotation matrix and the initial posture information.

[0125] In the above-mentioned embodiment provided by the present application, the electronic device includes an inertial measurement device, the first image and the second image correspond to different cameras in the camera module of the electronic device, the first image is captured at a first moment, and in the process of obtaining the first posture information of the first image, the initial posture information of the camera module is obtained; a first rotation matrix is ​​determined based on the output information of the inertial measurement device from the initial moment to the first moment and the output time interval of the inertial measurement device; and the first posture information is determined based on the first rotation matrix and the initial posture information. In this way, the real-time and accuracy of the determination of the first posture information is guaranteed, thereby improving the accuracy of subsequent image processing based on the first posture information.

[0126] In an embodiment of the present application, the processing unit 502 is specifically used to: determine a transformation matrix based on the first posture information and the second posture information; extract multiple first feature points in the first image and multiple second feature points in the second image, and the multiple first feature points and the multiple second feature points correspond one to one; construct an error function corresponding to each pair of first feature points and second feature points based on the transformation matrix; construct a target optimization function based on the multiple error functions; solve the target optimization function to obtain disturbance information.

[0127] In the above-mentioned embodiment provided by the present application, in the process of determining the disturbance information between the first image and the second image based on the first posture information and the second posture information, a transformation matrix is ​​determined based on the first posture information and the second posture information; multiple first feature points in the first image and multiple second feature points in the second image are extracted, with the multiple first feature points and the multiple second feature points corresponding one to one; an error function corresponding to each pair of first feature points and second feature points is constructed based on the transformation matrix; a target optimization function is constructed based on the multiple error functions; and the target optimization function is solved to obtain the disturbance information. In this way, the accuracy of determining the disturbance information is ensured, thereby improving the accuracy of subsequent image processing based on the disturbance information.

[0128] In the embodiment of the present application, the disturbance information includes a second rotation matrix and a translation matrix, and the processing unit 502 is specifically configured to determine the third posture information according to the product of the second posture information, the second rotation matrix and the translation matrix.

[0129] In the above-mentioned embodiment provided by the present application, the disturbance information includes a second rotation matrix and a translation matrix. When correcting the second posture information based on the disturbance information to obtain the third posture information, the third posture information is determined based on the product of the second posture information, the second rotation matrix, and the translation matrix. In this way, correcting the real-time posture information of the second image captured after the camera is switched can reduce the pose transformation and disturbance transformation between the first and second images during the camera switching process, thereby improving the quality of the image captured during the camera switching.

[0130] In the embodiment of the present application, after determining the disturbance information between the first image and the second image, the processing unit 502 is further configured to: determine a fourth image based on the first image and the disturbance information; compare the second image and the fourth image, and determine the accuracy of the disturbance information based on the comparison result; and adjust the disturbance information if the accuracy is less than a preset threshold.

[0131] In the above-described embodiment provided by this application, after determining the disturbance information between the first and second images, a fourth image is determined based on the first image and the disturbance information. The second and fourth images are compared, and the accuracy of the disturbance information is determined based on the comparison result. If the accuracy is less than a preset threshold, the disturbance information is adjusted. This ensures the accuracy of the obtained disturbance information, thereby improving the accuracy of subsequent image processing based on the disturbance information.

[0132] The image processing device 500 in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0133] The image processing device 500 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0134] The image processing device 500 provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 1 to 6. To avoid repetition, they are not described here.

[0135] Optionally, as shown in Figure 10, an embodiment of the present application also provides an electronic device 600, including a processor 602 and a memory 604, and the memory 604 stores a program or instruction that can be run on the processor 602. When the program or instruction is executed by the processor 602, the various steps of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0136] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0137] FIG11 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0138] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709 and a processor 710.

[0139] Those skilled in the art will appreciate that the electronic device 700 may further include a power source (e.g., a battery) to power various components. The power source may be logically connected to the processor 710 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG11 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.

[0140] The processor 710 is configured to obtain first posture information of the first image and second posture information of the second image.

[0141] The processor 710 is further configured to determine disturbance information between the first image and the second image based on the first posture information and the second posture information.

[0142] The processor 710 is further configured to correct the second posture information according to the disturbance information to obtain third posture information.

[0143] The processor 710 is further configured to perform smoothing processing on the third posture information to obtain fourth posture information.

[0144] The processor 710 is further configured to obtain a third image according to the fourth posture information and the second image.

[0145] In an embodiment of the present application, first posture information of a first image and second posture information of a second image are obtained; disturbance information between the first image and the second image is determined based on the first posture information and the second posture information; the second posture information is corrected based on the disturbance information to obtain third posture information; the third posture information is smoothed to obtain fourth posture information; and the third image is obtained based on the fourth posture information and the second image. In an embodiment of the present application, in the process of switching cameras for shooting, the multiple cameras in the camera module are regarded as a whole camera, and the switching of different cameras is regarded as inter-frame disturbance. Based on the posture information and disturbance information of different frame images, the picture taken after the camera is switched is processed, thereby processing the inter-frame disturbance caused by the camera switching. In this way, the problem that the camera of the electronic device has a significant jump in the picture taken at the moment of switching, thereby reducing the quality of the captured picture, is solved, and smooth switching of the camera is achieved. Moreover, during the camera switching process, only a single camera needs to be kept turned on at the same time, which reduces the power consumption of the electronic device.

[0146] Optionally, the electronic device includes an inertial measurement device, the first image and the second image correspond to different cameras in a camera module of the electronic device, the shooting time of the first image is a first moment, and the processor 710 is specifically used to: obtain initial posture information of the camera module; determine the first rotation matrix based on the output information of the inertial measurement device from the initial moment to the first moment and the output time interval of the inertial measurement device; determine the first posture information based on the first rotation matrix and the initial posture information.

[0147] In the above-mentioned embodiment provided by the present application, the electronic device includes an inertial measurement device, the first image and the second image correspond to different cameras in the camera module of the electronic device, the first image is captured at a first moment, and in the process of obtaining the first posture information of the first image, the initial posture information of the camera module is obtained; a first rotation matrix is ​​determined based on the output information of the inertial measurement device from the initial moment to the first moment and the output time interval of the inertial measurement device; and the first posture information is determined based on the first rotation matrix and the initial posture information. In this way, the real-time and accuracy of the determination of the first posture information is guaranteed, thereby improving the accuracy of subsequent image processing based on the first posture information.

[0148] Optionally, the processor 710 is specifically used to: determine a transformation matrix based on the first posture information and the second posture information; extract multiple first feature points in the first image and multiple second feature points in the second image, and the multiple first feature points and the multiple second feature points correspond one to one; construct an error function corresponding to each pair of first feature points and second feature points based on the transformation matrix; construct a target optimization function based on the multiple error functions; solve the target optimization function to obtain disturbance information.

[0149] In the above-mentioned embodiment provided by the present application, in the process of determining the disturbance information between the first image and the second image based on the first posture information and the second posture information, a transformation matrix is ​​determined based on the first posture information and the second posture information; multiple first feature points in the first image and multiple second feature points in the second image are extracted, with the multiple first feature points and the multiple second feature points corresponding one to one; an error function corresponding to each pair of first feature points and second feature points is constructed based on the transformation matrix; a target optimization function is constructed based on the multiple error functions; and the target optimization function is solved to obtain the disturbance information. In this way, the accuracy of determining the disturbance information is ensured, thereby improving the accuracy of subsequent image processing based on the disturbance information.

[0150] Optionally, the disturbance information includes a second rotation matrix and a translation matrix, and the processor 710 is specifically configured to determine third posture information according to the product of the second posture information, the second rotation matrix and the translation matrix.

[0151] In the above-mentioned embodiment provided by the present application, the disturbance information includes a second rotation matrix and a translation matrix. When correcting the second posture information based on the disturbance information to obtain the third posture information, the third posture information is determined based on the product of the second posture information, the second rotation matrix, and the translation matrix. In this way, correcting the real-time posture information of the second image captured after the camera is switched can reduce the pose transformation and disturbance transformation between the first and second images during the camera switching process, thereby improving the quality of the image captured during the camera switching.

[0152] Optionally, after determining the disturbance information between the first image and the second image, the processor 710 is further used to: determine a fourth image based on the first image and the disturbance information; compare the second image and the fourth image, and determine the accuracy of the disturbance information based on the comparison result; and adjust the disturbance information when the accuracy is less than a preset threshold.

[0153] In the above-described embodiment provided by this application, after determining the disturbance information between the first and second images, a fourth image is determined based on the first image and the disturbance information. The second and fourth images are compared, and the accuracy of the disturbance information is determined based on the comparison result. If the accuracy is less than a preset threshold, the disturbance information is adjusted. This ensures the accuracy of the obtained disturbance information, thereby improving the accuracy of subsequent image processing based on the disturbance information.

[0154] It should be understood that in an embodiment of the present application, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0155] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0156] Processor 710 may include one or more processing units. Optionally, processor 710 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 710.

[0157] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0158] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0159] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0160] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0161] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0162] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0164] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, performed by an electronic device, comprising: Acquire first posture information of the first image and second posture information of the second image; determining disturbance information between the first image and the second image according to the first posture information and the second posture information; Correcting the second posture information according to the disturbance information to obtain third posture information; performing smoothing processing on the third posture information to obtain fourth posture information; A third image is obtained according to the fourth posture information and the second image.

2. The image processing method according to claim 1, wherein: The electronic device includes an inertial measurement device, the first image and the second image are captured by different cameras in a camera module of the electronic device, the first image is captured at a first moment, and obtaining first posture information of the first image includes: Obtaining initial posture information of the camera module; determining a first rotation matrix according to output information of the inertial measurement device from an initial moment to the first moment and an output time interval of the inertial measurement device; The first posture information is determined according to the first rotation matrix and the initial posture information.

3. The image processing method according to claim 1, wherein: The determining, based on the first posture information and the second posture information, disturbance information between the first image and the second image includes: determining a transformation matrix according to the first posture information and the second posture information; Extracting a plurality of first feature points from the first image and a plurality of second feature points from the second image, wherein the plurality of first feature points and the plurality of second feature points correspond one to one; constructing an error function corresponding to each pair of the first feature point and the second feature point based on the transformation matrix; Constructing a target optimization function according to the plurality of error functions; Solve the target optimization function to obtain the disturbance information.

4. The image processing method according to claim 1, wherein: The disturbance information includes a second rotation matrix and a translation matrix, and the correcting the second posture information according to the disturbance information to obtain the third posture information includes: The third posture information is determined according to the product of the second posture information, the second rotation matrix, and the translation matrix.

5. The image processing method according to any one of claims 1 to 4, wherein: After determining the disturbance information between the first image and the second image, the image processing method further includes: determining a fourth image according to the first image and the disturbance information; comparing the second image and the fourth image, and determining an accuracy rate of the disturbance information according to a comparison result; When the accuracy is less than a preset threshold, the disturbance information is adjusted.

6. An image processing device, applied to an electronic device, comprising: a processing unit, configured to obtain first posture information of the first image and second posture information of the second image; The processing unit is further configured to determine disturbance information between the first image and the second image based on the first posture information and the second posture information; The processing unit is further configured to correct the second posture information according to the disturbance information to obtain third posture information; The processing unit is further configured to perform smoothing processing on the third posture information to obtain fourth posture information; The processing unit is further configured to obtain a third image based on the fourth posture information and the second image.

7. The image processing apparatus according to claim 6, wherein: The electronic device includes an inertial measurement device, the first image and the second image are captured by different cameras in a camera module of the electronic device, the first image is captured at a first moment, and the processing unit is specifically configured to: Obtaining initial posture information of the camera module; determining a first rotation matrix according to output information of the inertial measurement device from an initial moment to the first moment and an output time interval of the inertial measurement device; The first posture information is determined according to the first rotation matrix and the initial posture information.

8. The image processing apparatus according to claim 6, wherein: The processing unit is specifically configured to: determining a transformation matrix according to the first posture information and the second posture information; Extracting a plurality of first feature points from the first image and a plurality of second feature points from the second image, wherein the plurality of first feature points and the plurality of second feature points correspond one to one; constructing an error function corresponding to each pair of the first feature point and the second feature point based on the transformation matrix; Constructing a target optimization function according to the plurality of error functions; Solve the target optimization function to obtain the disturbance information.

9. The image processing apparatus according to claim 6, wherein: The disturbance information includes a second rotation matrix and a translation matrix, and the processing unit is specifically configured to: The third posture information is determined according to the product of the second posture information, the second rotation matrix, and the translation matrix.

10. The image processing apparatus according to any one of claims 6 to 9, wherein: After determining the disturbance information between the first image and the second image, the processing unit is further configured to: determining a fourth image according to the first image and the disturbance information; comparing the second image and the fourth image, and determining an accuracy rate of the disturbance information according to a comparison result; When the accuracy is less than a preset threshold, the disturbance information is adjusted.

11. An electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the method according to claims 1-5.

12. A readable storage medium storing a program or instruction, wherein the program or instruction is executed by a processor to implement the method according to claims 1-5.

13. A chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the method according to claims 1-5.

14. A computer program product, wherein the program product is executed by at least one processor to implement the method according to any one of claims 1 to 5.

Images